You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/02/14 08:39:51 UTC
[GitHub] [flink] infoverload opened a new pull request #18746: Dt/fix security pages
infoverload opened a new pull request #18746:
URL: https://github.com/apache/flink/pull/18746
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Fix security section of the docs
## Brief change log
- reorganization of flow
- add links and more text
- lots of restructuring and renaming things
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "112a4f6d047be628b5e0bc34ba54d246aea7f4c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "112a4f6d047be628b5e0bc34ba54d246aea7f4c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
* 112a4f6d047be628b5e0bc34ba54d246aea7f4c9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d46bd54e8d4e05e791d1679c5305100ec614c4c1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386)
* 91c09fb42686497a7e80405effd8fe474c24d4df Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc4c3c9c52616fe913872e5396bcef703c9ce45e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] knaufk commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
knaufk commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r809011344
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,292 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This guide describes how Flink security works in the context of various [deployment modes]({{< ref "docs/deployment/overview" >}}),
+[file systems]({{< ref "docs/deployment/filesystems/overview" >}}), [connectors]({{< ref "docs/connectors" >}}),
+and [state backends]({{< ref "docs/ops/state/state_backends" >}}).
+
+## Running a Secure Flink Cluster in Standalone Mode
+
+Standalone mode is the most barebone way of deploying a Flink cluster where Flink processes are launched
+as processes on the operating system. You can learn more [here]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on all cluster nodes.
+2. Ensure that the Kerberos keytab file exists at the path indicated by the `security.kerberos.login.keytab`
+ setting on all cluster nodes.
+3. Deploy the Flink cluster as normal.
+
+You need to have an appropriate Java keystore and truststore accessible from each node in the Flink
+cluster. For standalone setups, this means copying the files to each node, or adding them to a shared
+mounted directory. For externally facing REST endpoints, the common name or subject alternative names
+in the certificate should match the node's hostname and IP address.
+
+Keystores and truststores can be generated using the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html).
+
+## Running a Secure Flink Cluster in Kubernetes
+
+Flink’s native Kubernetes integration allows you to directly deploy Flink on a running Kubernetes cluster.
+You can learn more [here]({{< ref "docs/deployment/resource-providers/native_kubernetes" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on the client node.
+2. Ensure that the Kerberos keytab file exists at the path as indicated by the `security.kerberos.login.keytab`
+ on the client node.
+3. Deploy the Flink cluster as normal.
+
+### Setting up Kerberos
+
+When deploying Flink on Kubernetes, the Kerberos keytab file is automatically copied from the client
+node to the Flink containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file can be
+either fetched from the cluster environment or uploaded by Flink. In the latter case, you need to
+configure the `security.kerberos.krb5-conf.path` setting to indicate the path of the Kerberos configuration
+file and Flink will copy this file to its containers/pods.
+
+### Setting up keystores and truststores
+
+You need to have an appropriate Java keystore and truststore accessible from each node in the Flink
+cluster. For container-based setups, this means adding the keystore and truststore files to the container
+images. For externally facing REST endpoints, the common name or subject alternative names in the
+certificate should match the node's hostname and IP address.
+
+## Running a Secure Flink Cluster in YARN
+
+Flink services are submitted to YARN’s ResourceManager, which spawns containers on machines managed
+by YARN NodeManagers. Flink deploys its JobManager and TaskManager instances into such containers.
+You can learn more [here]({{< ref "docs/deployment/resource-providers/yarn" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on the client node.
+2. Ensure that the Kerberos keytab file exists at the path as indicated by the `security.kerberos.login.keytab`
+ on the client node.
+3. Deploy the Flink cluster as normal.
+
+### Setting up Kerberos
+
+When deploying Flink on YARN, the Kerberos keytab is automatically copied from the client node to the
+Flink containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file can be
+either fetched from the cluster environment or uploaded by Flink. In the latter case, you need to
+configure the `security.kerberos.krb5-conf.path` setting to indicate the path of the Kerberos configuration
+file and Flink will copy this file to its containers/pods.
+
+#### Using `kinit` and without keytabs
+
+[`kinit`](https://web.mit.edu/kerberos/krb5-1.12/doc/user/user_commands/kinit.html) is used to obtain
+and cache Kerberos ticket-granting tickets.
+
+With YARN, it is possible to deploy a secure Flink cluster without a Kerberos keytab, which avoids the
+complexity of generating a keytab and entrusting the cluster manager with it. In this scenario, the
+ticket-granting-ticket cache, which is managed by `kinit`, is used and the Flink CLI acquires Hadoop
+delegation tokens (for HDFS and for HBase) instead. The main disadvantage is that the cluster is necessarily
+short-lived since the generated delegation tokens will expire (typically within a week).
+
+These are the steps to run a secure Flink cluster using `kinit`:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on the client node.
+2. Log in using the `kinit` command.
+3. Deploy the Flink cluster as normal.
+
+### Setting up keystores and truststores
+
+You need to have an appropriate Java keystore and truststore accessible from each node in the Flink
+cluster. For YARN setups, the cluster deployment phase can automatically distribute the keystore and
+truststore files. For externally facing REST endpoints, the common name or subject alternative names in the
+certificate should match the node's hostname and IP address.
+
+For the externally facing REST endpoint, the common name or subject alternative names in the certificate
+should match the node's hostname and IP address.
+
+For more information, see the [documentation on YARN security](https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md).
+
+## Setting Up SSL
+
+The easiest way to set up SSL is to generate a dedicated public/private key pair and self-signed
+certificate for the Flink deployment. The keystore and truststore are identical and contains
+only that key pair and certificate.
+
+In an environment where operators are constrained to use firm-wide Internal Certificate Authorities
+(cannot generate self-signed certificates), we recommend to still have a dedicated key pair and certificate
+for the Flink deployment, signed by that CA. However, the truststore must then also contain the
+CA's public certificate to accept the deployment's certificate during the SSL handshake (a requirement
+in the JDK truststore implementation).
+
+{{< hint danger >}}
+It is critical that you specify the fingerprint of the deployment certificate (via the `security.ssl.internal.cert.fingerprint`
+setting), when it is not self-signed. This is to pin that certificate as the only trusted certificate
+and prevent the truststore from trusting all certificates signed by that CA.
+{{< /hint >}}
+
+{{< hint info >}}
+Since internal connections are mutually authenticated with shared certificates, Flink can skip
+hostname verification. This makes container-based setups easier.
+{{< /hint >}}
+
+### Tips for a YARN Deployment
+
+To secure the external REST endpoint, you need to issue the REST endpoint's certificate such that it
+is valid for all hosts that the JobManager may get deployed to. This can be done with a wildcard DNS
+name or by adding multiple DNS names.
+
+The easiest way to deploy keystores and truststore is via the YARN client's *ship files* option (`-yt`).
+Copy the keystore and truststore files into a local directory (i.e. `deploy-keys/`) and start the YARN
+session like this:
+
+```bash
+flink run -m yarn-cluster -yt deploy-keys/ flinkapp.jar
+```
+
+When deployed using YARN, Flink's web UI is accessible through the YARN proxy's "Tracking URL". To
+ensure that the YARN proxy is able to access Flink's HTTPS URL, you need to configure the YARN proxy
+to accept Flink's SSL certificates. To do this, add the custom CA certificate into Java's default
+truststore on the YARN proxy node.
+
+### Example SSL Setup for Standalone Deployment and Kubernetes Deployment
Review comment:
Is this really specific for standalone or kubernetes? If not, I'd say it rather belongs to the SSL page instead or we just link to an external guide for the non-Flink specific parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r811821655
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
+
+Kerberos can be used to authenticate connections to:
+- Hadoop and its components (YARN, HDFS, HBase)
+- ZooKeeper
+- Kafka (0.9+)
+
+The current implementation supports running Flink clusters (JobManager / TaskManager / Jobs) with two
+authentication modes:
+- a configured [Kerberos keytab](https://web.mit.edu/kerberos/krb5-devel/doc/basic/keytab_def.html) credential
+- [Hadoop delegation tokens](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html)
+
+In production deployments, streaming jobs usually run for long periods of time. It is important to be
+able to authenticate to secured data sources throughout the lifetime of the job. Kerberos keytabs are
+the preferred authentication approach because they won't expire during the lifetime of long-running
+stream processing applications, unlike a Hadoop delegation token or ticket cache entry.
+
+Note that the credentials are tied to a Flink cluster and not to a running job. Thus, all applications
+that run on the same cluster use the same authentication token and all jobs within a cluster will share
+the credentials configured for that cluster. If you need to work with different credentials, you should
+start a new cluster. For example, to use a different keytab for a certain job, simply launch a separate
+Flink cluster with a different configuration. Numerous Flink clusters may run side-by-side in a Kubernetes
+or YARN environment.
+
+Note that it is possible to enable and configure the use of Kerberos independently for each service
+or connector that is capable of being used with Kerberos. For example, you may enable Hadoop security
+without enabling the use of Kerberos for ZooKeeper, or vice versa.
+
+All services using Kerberos will use the same credentials. If you need to run some jobs with different
+Kerberos credentials, those jobs will have to run in a different cluster that is configured to use
+those other credentials. For example, you can decide to use Kerberos for Hadoop security, but not for ZooKeeper.
+
+## Using Kerberos with Flink Security Modules
+
+The internal architecture of Flink security is based on _security modules_ (which implements [`org.apache.flink.runtime.security.modules.SecurityModule`](https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/runtime/security/modules/SecurityModule.html)).
+These modules are installed at startup.
+
+### Hadoop Security Module
+
+This module uses the Hadoop `UserGroupInformation` (UGI) class to establish a process-wide *login user*
+context. The login user is then used for all interactions with Hadoop, including HDFS, HBase, and YARN.
+
+If Hadoop security is enabled (in `core-site.xml`), the login user will have whatever Kerberos credential
+is configured. Otherwise, the login user conveys only the user identity of the OS account that launched
+the cluster.
+
+### JAAS Security Module
+
+This module provides a dynamic [JAAS](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jaas/JAASRefGuide.html)
+configuration to the cluster, making available the configured Kerberos credential to ZooKeeper, Kafka,
+and other such components that rely on JAAS.
+
+Note that the user may also provide a static JAAS configuration file using the mechanisms described
+in the [Java SE Documentation](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html).
+Static entries override any dynamic entries provided by this module.
+
+### ZooKeeper Security Module
+
+This module configures certain process-wide ZooKeeper security-related settings, namely the ZooKeeper
+service name (default: `zookeeper`) and the JAAS login context name (default: `Client`).
+
+## Ticket Renewal
+
+A Ticket Granting Ticket (TGT) is a small, encrypted identification file with a limited validity period.
Review comment:
I will make some changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r839566228
##########
File path: docs/content/docs/security/overview.md
##########
@@ -0,0 +1,87 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL/TLS](https://www.ssl.com/faqs/faq-what-is-ssl/).
+
+{{< hint info >}}
+SSL and TLS are both encryption protocols for the transport layer, used for encrypting data between
+the client and server. The TLS protocol is the successor to SSL. Flink enforces TLS by default, but
+you can also switch back to SSL.
+{{< /hint >}}
+
+## What needs to be secured in Flink?
+
+When we talk about security for Flink, we generally make a distinction between securing the internal
+communication within the Flink cluster (i.e. between the TaskManagers, between the TaskManagers and
+the JobManager) and securing the external communication between the cluster and the outside world.
+
+Internally, [Netty](https://netty.io) is used for the TCP connections used for data exchange among
+the TaskManagers, and [Akka](https://akka.io) is used for RPC between the JobManager and the TaskManagers.
+
+Externally, HTTP is used for pretty much everything, except that some external services used as sources
+or sinks may use some other network protocol.
+
+## What is supported by Flink?
+
+Security enhancement features by the Flink community make it easy to access secured data, protect
+associated credentials, and increase overall security in a Flink cluster. The following security
+measures are currently supported:
+
+- Authentication of connections between Flink processes
+- Encryption of data transferred between Flink processes using SSL (Note that there is a performance
+ degradation when SSL is enabled, the magnitude of which depends on the CPU type, the JVM implementation,
+ and the key size.)
+- Authorization of read / write operations by clients
+- Authorization is pluggable and integration with external authorization services is supported
+
+It is worth noting that security needs to be manually configured because the overall philosophy in
+Flink is to have defaults that work out-of-the-box, without any configuration. Thus, non-secured
+clusters are supported, as well as a mix of authenticated, unauthenticated, encrypted and unencrypted
+clients.
+
+## Where are its flaws? How should they be addressed?
+
+Web UI unsecure
+not secure by default
+
+## What needs to be solved outside of Flink?
+
+connectors
Review comment:
this too 😬
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc4c3c9c52616fe913872e5396bcef703c9ce45e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720)
* 9f510145df68812e0eed11126fa839e85dedd26f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f510145df68812e0eed11126fa839e85dedd26f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542",
"triggerID" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"triggerType" : "PUSH"
}, {
"hash" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545",
"triggerID" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542)
* ec06532a2ea9edb00c82ce9304057966e5ef9ae7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r809209910
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
+
+Kerberos can be used to authenticate connections to:
+- Hadoop and its components (YARN, HDFS, HBase)
+- ZooKeeper
+- Kafka (0.9+)
+
+The current implementation supports running Flink clusters (JobManager / TaskManager / Jobs) with two
+authentication modes:
+- a configured [Kerberos keytab](https://web.mit.edu/kerberos/krb5-devel/doc/basic/keytab_def.html) credential
+- [Hadoop delegation tokens](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html)
+
+In production deployments, streaming jobs usually run for long periods of time. It is important to be
+able to authenticate to secured data sources throughout the lifetime of the job. Kerberos keytabs are
+the preferred authentication approach because they won't expire during the lifetime of long-running
+stream processing applications, unlike a Hadoop delegation token or ticket cache entry.
+
+Note that the credentials are tied to a Flink cluster and not to a running job. Thus, all applications
+that run on the same cluster use the same authentication token and all jobs within a cluster will share
+the credentials configured for that cluster. If you need to work with different credentials, you should
+start a new cluster. For example, to use a different keytab for a certain job, simply launch a separate
+Flink cluster with a different configuration. Numerous Flink clusters may run side-by-side in a Kubernetes
+or YARN environment.
+
+Note that it is possible to enable and configure the use of Kerberos independently for each service
+or connector that is capable of being used with Kerberos. For example, you may enable Hadoop security
+without enabling the use of Kerberos for ZooKeeper, or vice versa.
+
+All services using Kerberos will use the same credentials. If you need to run some jobs with different
+Kerberos credentials, those jobs will have to run in a different cluster that is configured to use
+those other credentials. For example, you can decide to use Kerberos for Hadoop security, but not for ZooKeeper.
+
+## Using Kerberos with Flink Security Modules
+
+The internal architecture of Flink security is based on _security modules_ (which implements [`org.apache.flink.runtime.security.modules.SecurityModule`](https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/runtime/security/modules/SecurityModule.html)).
+These modules are installed at startup.
+
+### Hadoop Security Module
+
+This module uses the Hadoop `UserGroupInformation` (UGI) class to establish a process-wide *login user*
+context. The login user is then used for all interactions with Hadoop, including HDFS, HBase, and YARN.
+
+If Hadoop security is enabled (in `core-site.xml`), the login user will have whatever Kerberos credential
+is configured. Otherwise, the login user conveys only the user identity of the OS account that launched
+the cluster.
+
+### JAAS Security Module
+
+This module provides a dynamic [JAAS](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jaas/JAASRefGuide.html)
+configuration to the cluster, making available the configured Kerberos credential to ZooKeeper, Kafka,
+and other such components that rely on JAAS.
+
+Note that the user may also provide a static JAAS configuration file using the mechanisms described
+in the [Java SE Documentation](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html).
+Static entries override any dynamic entries provided by this module.
+
+### ZooKeeper Security Module
+
+This module configures certain process-wide ZooKeeper security-related settings, namely the ZooKeeper
+service name (default: `zookeeper`) and the JAAS login context name (default: `Client`).
+
+## Ticket Renewal
+
+A Ticket Granting Ticket (TGT) is a small, encrypted identification file with a limited validity period.
Review comment:
I got the information from here: https://en.wikipedia.org/wiki/Ticket_Granting_Ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d46bd54e8d4e05e791d1679c5305100ec614c4c1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542",
"triggerID" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"triggerType" : "PUSH"
}, {
"hash" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
* bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542)
* ec06532a2ea9edb00c82ce9304057966e5ef9ae7 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f510145df68812e0eed11126fa839e85dedd26f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055)
* 8287d2291eb35027698751ab3a6519669ce59da9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
* bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542",
"triggerID" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"triggerType" : "PUSH"
}, {
"hash" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545",
"triggerID" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"triggerType" : "PUSH"
}, {
"hash" : "04b63296c8bd42f3191643ccc2990578c215e593",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33154",
"triggerID" : "04b63296c8bd42f3191643ccc2990578c215e593",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 04b63296c8bd42f3191643ccc2990578c215e593 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33154)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542",
"triggerID" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"triggerType" : "PUSH"
}, {
"hash" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545",
"triggerID" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"triggerType" : "PUSH"
}, {
"hash" : "04b63296c8bd42f3191643ccc2990578c215e593",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33154",
"triggerID" : "04b63296c8bd42f3191643ccc2990578c215e593",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec06532a2ea9edb00c82ce9304057966e5ef9ae7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545)
* 04b63296c8bd42f3191643ccc2990578c215e593 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33154)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r839565867
##########
File path: docs/content/docs/security/overview.md
##########
@@ -0,0 +1,87 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL/TLS](https://www.ssl.com/faqs/faq-what-is-ssl/).
+
+{{< hint info >}}
+SSL and TLS are both encryption protocols for the transport layer, used for encrypting data between
+the client and server. The TLS protocol is the successor to SSL. Flink enforces TLS by default, but
+you can also switch back to SSL.
+{{< /hint >}}
+
+## What needs to be secured in Flink?
+
+When we talk about security for Flink, we generally make a distinction between securing the internal
+communication within the Flink cluster (i.e. between the TaskManagers, between the TaskManagers and
+the JobManager) and securing the external communication between the cluster and the outside world.
+
+Internally, [Netty](https://netty.io) is used for the TCP connections used for data exchange among
+the TaskManagers, and [Akka](https://akka.io) is used for RPC between the JobManager and the TaskManagers.
+
+Externally, HTTP is used for pretty much everything, except that some external services used as sources
+or sinks may use some other network protocol.
+
+## What is supported by Flink?
+
+Security enhancement features by the Flink community make it easy to access secured data, protect
+associated credentials, and increase overall security in a Flink cluster. The following security
+measures are currently supported:
+
+- Authentication of connections between Flink processes
+- Encryption of data transferred between Flink processes using SSL (Note that there is a performance
+ degradation when SSL is enabled, the magnitude of which depends on the CPU type, the JVM implementation,
+ and the key size.)
+- Authorization of read / write operations by clients
+- Authorization is pluggable and integration with external authorization services is supported
+
+It is worth noting that security needs to be manually configured because the overall philosophy in
+Flink is to have defaults that work out-of-the-box, without any configuration. Thus, non-secured
+clusters are supported, as well as a mix of authenticated, unauthenticated, encrypted and unencrypted
+clients.
+
+## Where are its flaws? How should they be addressed?
+
+Web UI unsecure
Review comment:
sorry I committed some stuff that I was just braindumping 😬
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
rmetzger commented on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1041269302
Since https://issues.apache.org/jira/browse/FLINK-24474, the one thing I wanted to mention in the docs seems to be resolved ;)
So in the future https://flink.apache.org/security.html will just have a link to the FAQ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r809116052
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
Review comment:
Because that is what it originally said in the docs. Is this not true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
zentol commented on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038881374
Please file a ticket in JIRA for larger changes to the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r811877267
##########
File path: docs/content/docs/deployment/security/overview.md
##########
@@ -0,0 +1,68 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL](https://www.ssl.com/faqs/faq-what-is-ssl/).
Review comment:
Ah, just found it. TLSv1.2 is the defa so we're fine here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91c09fb42686497a7e80405effd8fe474c24d4df Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444)
* c4f76a841937d005508c859a643c6ac4c0fcbf7f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r807711902
##########
File path: docs/content/docs/deployment/security/ssl.md
##########
@@ -0,0 +1,243 @@
+---
+title: "Encryption and Authentication using SSL"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster](#).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+Most of these connections are exposed via REST/HTTP endpoints (and used by the web UI). Some external
+services used as sources or sinks may use some other network protocol.
+
+The server will, by default, accept connections from any client, meaning that the REST endpoint does
+not authenticate the client. These REST endpoints, however, can be configured to require SSL encryption
+and mutual authentication.
+
+However, the recommended approach is setting up and configuring a dedicated proxy service (a "sidecar
+proxy") that controls access to the REST endpoint. This involves binding the REST endpoint to the
+loopback interface (or the pod-local interface in Kubernetes) and starting a REST proxy that authenticates
+and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/)
+or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html).
+
+The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety
+of authentication options and thus better integration into existing infrastructures.
+
+## Queryable State
+
+Connections to the [queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}})
+endpoints is currently not authenticated or encrypted.
+
+## SSL Setups
+
+{{< img src="/fig/ssl_mutual_auth.svg" alt="SSL Mutual Authentication" width=75% >}}
+
+Each participant has a keystore and a truststore, which are files.
+
+A keystore contains a certificate (which contains a public key) and a private key. A truststore
+contains trusted certificates and certificate chains/authorities.
+
+Establishing encrypted, authenticated communication is a multi-step process, shown in the figure.
+Certificates are exchanged and validated against the truststore, after which the two parties can
+safely communicate.
+
+### Typical SSL Setup in Flink
+
+For mutually authenticated internal connections, note that:
+
+- a keystore and a truststore can contain the same dedicated certificate
+- the same file can be used for both keystore and truststore
+- wildcard hostnames or addresses can be used
+
+For internal communication between servers in a Flink cluster, a secure setup can be established with
+a single, self-signed certificate that all parties use as both their keystore and truststore. You can
+also use this approach for external communication when establishing mutual authentication for communication
+between clients and the Flink Master.
+
+### Configuring Keystores and Truststores
+
+The SSL configuration requires configuring a keystore and a truststore such that the truststore trusts
+the keystore's certificate.
+
+You can use the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html)
+to generate keys, certificates, keystores, and truststores:
+
+```bash
+ keytool -genkeypair -alias flink.internal -keystore internal.keystore \
+ -dname "CN=flink.internal" -storepass internal_store_password -keyalg RSA \
+ -keysize 4096 -storetype PKCS12
+```
+
+| Deployment mode | How to add the files |
+|------------------------|-------------------------------------------------------------------------|
+| Standalone clusters | copy the files to each node, or add them to a shared mounted filesystem |
+| Containerized clusters | add the files to the container images |
+| YARN | the cluster deployment phase can distribute these files |
+
+### Using Cipher Suites
+
+While the acts of encryption and decryption themselves are performed by keys, cipher suites outline
+the set of steps that the keys must follow to do so and the order in which these steps are executed.
+There are numerous cipher suites out there, each one with varying instructions on the encryption and
+decryption process.
+
+{{< hint warning >}}
+The [IETF RFC 7525](https://tools.ietf.org/html/rfc7525) recommends using a specific set of cipher
+suites for strong security. Since these cipher suites are not available on many setups out-of-the-box,
+Flink defaults to TLS_RSA_WITH_AES_128_CBC_SHA (a slightly weaker but more widely available cipher suite).
Review comment:
Well, this is compatible but weak in many ways. Including but not limited to:
* In 2013, researchers demonstrated a timing attack against several TLS implementations using the CBC encryption algorithm (see [isg.rhul.ac.uk](http://www.isg.rhul.ac.uk/tls/Lucky13.html)). Additionally, the CBC mode is vulnerable to plain-text attacks in TLS 1.0, SSL 3.0 and lower. A fix has been introduced with TLS 1.2 in form of the GCM mode which is not vulnerable to the BEAST attack. GCM should be preferred over CBC.
* The Secure Hash Algorithm 1 has been proven to be insecure as of 2017 (see [shattered.io](https://shattered.io/)).
Defaulting to a weak suite makes users think that they're safe (which is not true) since we broadcast out-of-the-box configuration. Choosing a better suite requires all users to read this and act accordingly. If we can't come up w/ a better suite then maybe a warning can be printed that this is the default and weak...
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
+
+Kerberos can be used to authenticate connections to:
+- Hadoop and its components (YARN, HDFS, HBase)
+- ZooKeeper
+- Kafka (0.9+)
+
+The current implementation supports running Flink clusters (JobManager / TaskManager / Jobs) with two
+authentication modes:
+- a configured [Kerberos keytab](https://web.mit.edu/kerberos/krb5-devel/doc/basic/keytab_def.html) credential
+- [Hadoop delegation tokens](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html)
+
+In production deployments, streaming jobs usually run for long periods of time. It is important to be
+able to authenticate to secured data sources throughout the lifetime of the job. Kerberos keytabs are
+the preferred authentication approach because they won't expire during the lifetime of long-running
+stream processing applications, unlike a Hadoop delegation token or ticket cache entry.
+
+Note that the credentials are tied to a Flink cluster and not to a running job. Thus, all applications
+that run on the same cluster use the same authentication token and all jobs within a cluster will share
+the credentials configured for that cluster. If you need to work with different credentials, you should
+start a new cluster. For example, to use a different keytab for a certain job, simply launch a separate
+Flink cluster with a different configuration. Numerous Flink clusters may run side-by-side in a Kubernetes
+or YARN environment.
+
+Note that it is possible to enable and configure the use of Kerberos independently for each service
+or connector that is capable of being used with Kerberos. For example, you may enable Hadoop security
+without enabling the use of Kerberos for ZooKeeper, or vice versa.
+
+All services using Kerberos will use the same credentials. If you need to run some jobs with different
Review comment:
I don't want to be peckish so we can have it like that but there are edge cases where this is not true.
Like Kafka delegation token can be provided from another subject and such case the provided cluster credentials are not used. Anyway, this is good as-is because that's more like a hack.
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
+
+Kerberos can be used to authenticate connections to:
+- Hadoop and its components (YARN, HDFS, HBase)
+- ZooKeeper
+- Kafka (0.9+)
+
+The current implementation supports running Flink clusters (JobManager / TaskManager / Jobs) with two
+authentication modes:
+- a configured [Kerberos keytab](https://web.mit.edu/kerberos/krb5-devel/doc/basic/keytab_def.html) credential
+- [Hadoop delegation tokens](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html)
+
+In production deployments, streaming jobs usually run for long periods of time. It is important to be
+able to authenticate to secured data sources throughout the lifetime of the job. Kerberos keytabs are
+the preferred authentication approach because they won't expire during the lifetime of long-running
Review comment:
This is partially true. Keytabs are subject to any password expiration policies that may be imposed on a principal. Thus, if a principal's password expires (or the password is changed), a keytab generated using that password will be rendered invalid.
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
+
+Kerberos can be used to authenticate connections to:
+- Hadoop and its components (YARN, HDFS, HBase)
+- ZooKeeper
+- Kafka (0.9+)
+
+The current implementation supports running Flink clusters (JobManager / TaskManager / Jobs) with two
+authentication modes:
+- a configured [Kerberos keytab](https://web.mit.edu/kerberos/krb5-devel/doc/basic/keytab_def.html) credential
+- [Hadoop delegation tokens](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html)
+
+In production deployments, streaming jobs usually run for long periods of time. It is important to be
+able to authenticate to secured data sources throughout the lifetime of the job. Kerberos keytabs are
+the preferred authentication approach because they won't expire during the lifetime of long-running
+stream processing applications, unlike a Hadoop delegation token or ticket cache entry.
+
+Note that the credentials are tied to a Flink cluster and not to a running job. Thus, all applications
+that run on the same cluster use the same authentication token and all jobs within a cluster will share
+the credentials configured for that cluster. If you need to work with different credentials, you should
+start a new cluster. For example, to use a different keytab for a certain job, simply launch a separate
+Flink cluster with a different configuration. Numerous Flink clusters may run side-by-side in a Kubernetes
+or YARN environment.
+
+Note that it is possible to enable and configure the use of Kerberos independently for each service
+or connector that is capable of being used with Kerberos. For example, you may enable Hadoop security
+without enabling the use of Kerberos for ZooKeeper, or vice versa.
+
+All services using Kerberos will use the same credentials. If you need to run some jobs with different
+Kerberos credentials, those jobs will have to run in a different cluster that is configured to use
+those other credentials. For example, you can decide to use Kerberos for Hadoop security, but not for ZooKeeper.
+
+## Using Kerberos with Flink Security Modules
+
+The internal architecture of Flink security is based on _security modules_ (which implements [`org.apache.flink.runtime.security.modules.SecurityModule`](https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/runtime/security/modules/SecurityModule.html)).
+These modules are installed at startup.
+
+### Hadoop Security Module
+
+This module uses the Hadoop `UserGroupInformation` (UGI) class to establish a process-wide *login user*
+context. The login user is then used for all interactions with Hadoop, including HDFS, HBase, and YARN.
+
+If Hadoop security is enabled (in `core-site.xml`), the login user will have whatever Kerberos credential
+is configured. Otherwise, the login user conveys only the user identity of the OS account that launched
+the cluster.
+
+### JAAS Security Module
+
+This module provides a dynamic [JAAS](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jaas/JAASRefGuide.html)
Review comment:
Nit: Is there a specific reason why we use sometimes java 8 and sometimes 7 links (like couple of lines further).
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
+
+Kerberos can be used to authenticate connections to:
+- Hadoop and its components (YARN, HDFS, HBase)
+- ZooKeeper
+- Kafka (0.9+)
+
+The current implementation supports running Flink clusters (JobManager / TaskManager / Jobs) with two
+authentication modes:
+- a configured [Kerberos keytab](https://web.mit.edu/kerberos/krb5-devel/doc/basic/keytab_def.html) credential
+- [Hadoop delegation tokens](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html)
+
+In production deployments, streaming jobs usually run for long periods of time. It is important to be
+able to authenticate to secured data sources throughout the lifetime of the job. Kerberos keytabs are
+the preferred authentication approach because they won't expire during the lifetime of long-running
+stream processing applications, unlike a Hadoop delegation token or ticket cache entry.
+
+Note that the credentials are tied to a Flink cluster and not to a running job. Thus, all applications
+that run on the same cluster use the same authentication token and all jobs within a cluster will share
+the credentials configured for that cluster. If you need to work with different credentials, you should
+start a new cluster. For example, to use a different keytab for a certain job, simply launch a separate
+Flink cluster with a different configuration. Numerous Flink clusters may run side-by-side in a Kubernetes
+or YARN environment.
+
+Note that it is possible to enable and configure the use of Kerberos independently for each service
+or connector that is capable of being used with Kerberos. For example, you may enable Hadoop security
+without enabling the use of Kerberos for ZooKeeper, or vice versa.
+
+All services using Kerberos will use the same credentials. If you need to run some jobs with different
+Kerberos credentials, those jobs will have to run in a different cluster that is configured to use
+those other credentials. For example, you can decide to use Kerberos for Hadoop security, but not for ZooKeeper.
+
+## Using Kerberos with Flink Security Modules
+
+The internal architecture of Flink security is based on _security modules_ (which implements [`org.apache.flink.runtime.security.modules.SecurityModule`](https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/runtime/security/modules/SecurityModule.html)).
+These modules are installed at startup.
+
+### Hadoop Security Module
+
+This module uses the Hadoop `UserGroupInformation` (UGI) class to establish a process-wide *login user*
+context. The login user is then used for all interactions with Hadoop, including HDFS, HBase, and YARN.
+
+If Hadoop security is enabled (in `core-site.xml`), the login user will have whatever Kerberos credential
+is configured. Otherwise, the login user conveys only the user identity of the OS account that launched
+the cluster.
+
+### JAAS Security Module
+
+This module provides a dynamic [JAAS](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jaas/JAASRefGuide.html)
+configuration to the cluster, making available the configured Kerberos credential to ZooKeeper, Kafka,
+and other such components that rely on JAAS.
+
+Note that the user may also provide a static JAAS configuration file using the mechanisms described
+in the [Java SE Documentation](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html).
+Static entries override any dynamic entries provided by this module.
+
+### ZooKeeper Security Module
+
+This module configures certain process-wide ZooKeeper security-related settings, namely the ZooKeeper
+service name (default: `zookeeper`) and the JAAS login context name (default: `Client`).
+
+## Ticket Renewal
+
+A Ticket Granting Ticket (TGT) is a small, encrypted identification file with a limited validity period.
Review comment:
TGT is simply not a file. A ticket cache may be file but can be keyring, api, etc... Of course Flink supports only file format of ticket cache.
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
+
+Kerberos can be used to authenticate connections to:
+- Hadoop and its components (YARN, HDFS, HBase)
+- ZooKeeper
+- Kafka (0.9+)
+
+The current implementation supports running Flink clusters (JobManager / TaskManager / Jobs) with two
+authentication modes:
+- a configured [Kerberos keytab](https://web.mit.edu/kerberos/krb5-devel/doc/basic/keytab_def.html) credential
+- [Hadoop delegation tokens](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html)
+
+In production deployments, streaming jobs usually run for long periods of time. It is important to be
+able to authenticate to secured data sources throughout the lifetime of the job. Kerberos keytabs are
+the preferred authentication approach because they won't expire during the lifetime of long-running
+stream processing applications, unlike a Hadoop delegation token or ticket cache entry.
+
+Note that the credentials are tied to a Flink cluster and not to a running job. Thus, all applications
+that run on the same cluster use the same authentication token and all jobs within a cluster will share
+the credentials configured for that cluster. If you need to work with different credentials, you should
+start a new cluster. For example, to use a different keytab for a certain job, simply launch a separate
+Flink cluster with a different configuration. Numerous Flink clusters may run side-by-side in a Kubernetes
+or YARN environment.
+
+Note that it is possible to enable and configure the use of Kerberos independently for each service
+or connector that is capable of being used with Kerberos. For example, you may enable Hadoop security
+without enabling the use of Kerberos for ZooKeeper, or vice versa.
+
+All services using Kerberos will use the same credentials. If you need to run some jobs with different
+Kerberos credentials, those jobs will have to run in a different cluster that is configured to use
+those other credentials. For example, you can decide to use Kerberos for Hadoop security, but not for ZooKeeper.
+
+## Using Kerberos with Flink Security Modules
+
+The internal architecture of Flink security is based on _security modules_ (which implements [`org.apache.flink.runtime.security.modules.SecurityModule`](https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/runtime/security/modules/SecurityModule.html)).
+These modules are installed at startup.
+
+### Hadoop Security Module
+
+This module uses the Hadoop `UserGroupInformation` (UGI) class to establish a process-wide *login user*
+context. The login user is then used for all interactions with Hadoop, including HDFS, HBase, and YARN.
+
+If Hadoop security is enabled (in `core-site.xml`), the login user will have whatever Kerberos credential
+is configured. Otherwise, the login user conveys only the user identity of the OS account that launched
+the cluster.
+
+### JAAS Security Module
+
+This module provides a dynamic [JAAS](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jaas/JAASRefGuide.html)
+configuration to the cluster, making available the configured Kerberos credential to ZooKeeper, Kafka,
+and other such components that rely on JAAS.
+
+Note that the user may also provide a static JAAS configuration file using the mechanisms described
+in the [Java SE Documentation](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html).
+Static entries override any dynamic entries provided by this module.
+
+### ZooKeeper Security Module
+
+This module configures certain process-wide ZooKeeper security-related settings, namely the ZooKeeper
+service name (default: `zookeeper`) and the JAAS login context name (default: `Client`).
+
+## Ticket Renewal
+
+A Ticket Granting Ticket (TGT) is a small, encrypted identification file with a limited validity period.
+The TGT file contains the session key, its expiration date, and the user's IP address.
Review comment:
Just for my own understanding what do we mean under: `user's IP address`?
I would understand it better what we mean here if you can point to the field in the fileformat here: https://web.mit.edu/kerberos/krb5-devel/doc/formats/ccache_file_format.html
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,231 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This document briefly describes how Flink security works in the context of various deployment
+mechanisms (Standalone, native Kubernetes, YARN), filesystems, connectors, and state backends.
+
+## Deployment Modes
+
+Here is some information specific to each deployment mode.
+
+### Standalone Mode
+
+Steps to run a secure Flink cluster in standalone/cluster mode:
+
+1. Add security-related configuration options to the Flink configuration file (on all cluster nodes)
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path indicated by `security.kerberos.login.keytab` on
+ all cluster nodes.
+3. Deploy Flink cluster as normal.
+
+### Native Kubernetes and YARN Mode
+
+Steps to run a secure Flink cluster in native Kubernetes and YARN mode:
+
+1. Add security-related configuration options to the Flink configuration file on the client
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path as indicated by `security.kerberos.login.keytab` on
+ the client node.
+3. Deploy Flink cluster as normal.
+
+In YARN and native Kubernetes mode, the keytab is automatically copied from the client to the Flink
+containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file can be
+either fetched from the cluster environment or uploaded by Flink. In the latter case, you need to
Review comment:
How can I imagine this in real life: `fetched from the cluster environment`?
Some users are let's say hard-coded `krb5.conf` files on cluster nodes. Is that what we mean here?
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,231 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This document briefly describes how Flink security works in the context of various deployment
+mechanisms (Standalone, native Kubernetes, YARN), filesystems, connectors, and state backends.
+
+## Deployment Modes
+
+Here is some information specific to each deployment mode.
+
+### Standalone Mode
+
+Steps to run a secure Flink cluster in standalone/cluster mode:
+
+1. Add security-related configuration options to the Flink configuration file (on all cluster nodes)
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path indicated by `security.kerberos.login.keytab` on
+ all cluster nodes.
+3. Deploy Flink cluster as normal.
+
+### Native Kubernetes and YARN Mode
+
+Steps to run a secure Flink cluster in native Kubernetes and YARN mode:
+
+1. Add security-related configuration options to the Flink configuration file on the client
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path as indicated by `security.kerberos.login.keytab` on
+ the client node.
+3. Deploy Flink cluster as normal.
+
+In YARN and native Kubernetes mode, the keytab is automatically copied from the client to the Flink
+containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file can be
+either fetched from the cluster environment or uploaded by Flink. In the latter case, you need to
+configure the `security.kerberos.krb5-conf.path` to indicate the path of the Kerberos configuration
+file and Flink will copy this file to its containers/pods.
+
+For more information, see the [documentation on YARN security](https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md).
+
+#### Using `kinit` (YARN only)
+
+In YARN mode, it is possible to deploy a secure Flink cluster without a keytab, using only the ticket
+cache (as managed by `kinit`). This avoids the complexity of generating a keytab and avoids entrusting
+the cluster manager with it. In this scenario, the Flink CLI acquires Hadoop delegation tokens (for
+HDFS and for HBase). The main drawback is that the cluster is necessarily short-lived since the generated
+delegation tokens will expire (typically within a week).
+
+Steps to run a secure Flink cluster using `kinit`:
+
+1. Add security-related configuration options to the Flink configuration file on the client
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Login using the `kinit` command.
+3. Deploy Flink cluster as normal.
+
+
+## SSL - Tips for YARN Deployment
+
+For YARN, you can use the tools of Yarn to help:
+
+- Configuring security for internal communication is exactly the same as in the example above.
+
+- To secure the REST endpoint, you need to issue the REST endpoint's certificate such that it is
+ valid for all hosts that the JobManager may get deployed to. This can be done with a wild card
+ DNS name, or by adding multiple DNS names.
+
+- The easiest way to deploy keystores and truststore is by YARN client's *ship files* option (`-yt`).
+ Copy the keystore and truststore files into a local directory (say `deploy-keys/`) and start the
+ YARN session as follows: `flink run -m yarn-cluster -yt deploy-keys/ flinkapp.jar`
+
+- When deployed using YARN, Flink's web dashboard is accessible through YARN proxy's Tracking URL.
+ To ensure that the YARN proxy is able to access Flink's HTTPS URL, you need to configure YARN proxy
+ to accept Flink's SSL certificates.
+ For that, add the custom CA certificate into Java's default truststore on the YARN Proxy node.
+
+
+## Creating and Deploying Keystores and Truststores
+
+Keys, Certificates, and the Keystores and Truststores can be generated using the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html).
+You need to have an appropriate Java Keystore and Truststore accessible from each node in the Flink cluster.
+
+- For standalone setups, this means copying the files to each node, or adding them to a shared mounted directory.
+- For container based setups, add the keystore and truststore files to the container images.
+- For Yarn setups, the cluster deployment phase can automatically distribute the keystore and truststore files.
+
+For the externally facing REST endpoint, the common name or subject alternative names in the certificate
+should match the node's hostname and IP address.
+
+## Example SSL Setup Standalone and Kubernetes
+
+**Internal Connectivity**
+
+Execute the following keytool commands to create a key pair in a keystore:
+
+```bash
+$ keytool -genkeypair \
+ -alias flink.internal \
+ -keystore internal.keystore \
+ -dname "CN=flink.internal" \
+ -storepass internal_store_password \
+ -keyalg RSA \
+ -keysize 4096 \
+ -storetype PKCS12
+```
+
+The single key/certificate in the keystore is used the same way by the server and client endpoints
+(mutual authentication). The key pair acts as the shared secret for internal security, and we can
+directly use it as keystore and truststore.
+
+```yaml
+security.ssl.internal.enabled: true
+security.ssl.internal.keystore: /path/to/flink/conf/internal.keystore
+security.ssl.internal.truststore: /path/to/flink/conf/internal.keystore
+security.ssl.internal.keystore-password: internal_store_password
Review comment:
Nit: In order to increase the quality of the doc maybe we can mention in a central place that storing plaintext passwords in config files is weak. Better way is either to use K8S secrets or environment variables.
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,231 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This document briefly describes how Flink security works in the context of various deployment
+mechanisms (Standalone, native Kubernetes, YARN), filesystems, connectors, and state backends.
+
+## Deployment Modes
+
+Here is some information specific to each deployment mode.
+
+### Standalone Mode
+
+Steps to run a secure Flink cluster in standalone/cluster mode:
+
+1. Add security-related configuration options to the Flink configuration file (on all cluster nodes)
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path indicated by `security.kerberos.login.keytab` on
+ all cluster nodes.
+3. Deploy Flink cluster as normal.
+
+### Native Kubernetes and YARN Mode
+
+Steps to run a secure Flink cluster in native Kubernetes and YARN mode:
+
+1. Add security-related configuration options to the Flink configuration file on the client
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path as indicated by `security.kerberos.login.keytab` on
+ the client node.
+3. Deploy Flink cluster as normal.
+
+In YARN and native Kubernetes mode, the keytab is automatically copied from the client to the Flink
+containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file can be
+either fetched from the cluster environment or uploaded by Flink. In the latter case, you need to
+configure the `security.kerberos.krb5-conf.path` to indicate the path of the Kerberos configuration
+file and Flink will copy this file to its containers/pods.
+
+For more information, see the [documentation on YARN security](https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md).
+
+#### Using `kinit` (YARN only)
+
+In YARN mode, it is possible to deploy a secure Flink cluster without a keytab, using only the ticket
+cache (as managed by `kinit`). This avoids the complexity of generating a keytab and avoids entrusting
+the cluster manager with it. In this scenario, the Flink CLI acquires Hadoop delegation tokens (for
+HDFS and for HBase). The main drawback is that the cluster is necessarily short-lived since the generated
+delegation tokens will expire (typically within a week).
+
+Steps to run a secure Flink cluster using `kinit`:
+
+1. Add security-related configuration options to the Flink configuration file on the client
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Login using the `kinit` command.
+3. Deploy Flink cluster as normal.
+
+
+## SSL - Tips for YARN Deployment
+
+For YARN, you can use the tools of Yarn to help:
+
+- Configuring security for internal communication is exactly the same as in the example above.
+
+- To secure the REST endpoint, you need to issue the REST endpoint's certificate such that it is
+ valid for all hosts that the JobManager may get deployed to. This can be done with a wild card
+ DNS name, or by adding multiple DNS names.
+
+- The easiest way to deploy keystores and truststore is by YARN client's *ship files* option (`-yt`).
+ Copy the keystore and truststore files into a local directory (say `deploy-keys/`) and start the
+ YARN session as follows: `flink run -m yarn-cluster -yt deploy-keys/ flinkapp.jar`
+
+- When deployed using YARN, Flink's web dashboard is accessible through YARN proxy's Tracking URL.
+ To ensure that the YARN proxy is able to access Flink's HTTPS URL, you need to configure YARN proxy
+ to accept Flink's SSL certificates.
+ For that, add the custom CA certificate into Java's default truststore on the YARN Proxy node.
+
+
+## Creating and Deploying Keystores and Truststores
+
+Keys, Certificates, and the Keystores and Truststores can be generated using the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html).
+You need to have an appropriate Java Keystore and Truststore accessible from each node in the Flink cluster.
+
+- For standalone setups, this means copying the files to each node, or adding them to a shared mounted directory.
+- For container based setups, add the keystore and truststore files to the container images.
+- For Yarn setups, the cluster deployment phase can automatically distribute the keystore and truststore files.
+
+For the externally facing REST endpoint, the common name or subject alternative names in the certificate
+should match the node's hostname and IP address.
+
+## Example SSL Setup Standalone and Kubernetes
+
+**Internal Connectivity**
+
+Execute the following keytool commands to create a key pair in a keystore:
+
+```bash
+$ keytool -genkeypair \
+ -alias flink.internal \
+ -keystore internal.keystore \
+ -dname "CN=flink.internal" \
+ -storepass internal_store_password \
+ -keyalg RSA \
+ -keysize 4096 \
+ -storetype PKCS12
+```
+
+The single key/certificate in the keystore is used the same way by the server and client endpoints
+(mutual authentication). The key pair acts as the shared secret for internal security, and we can
+directly use it as keystore and truststore.
+
+```yaml
+security.ssl.internal.enabled: true
+security.ssl.internal.keystore: /path/to/flink/conf/internal.keystore
+security.ssl.internal.truststore: /path/to/flink/conf/internal.keystore
+security.ssl.internal.keystore-password: internal_store_password
+security.ssl.internal.truststore-password: internal_store_password
+security.ssl.internal.key-password: internal_store_password
+```
+
+**REST Endpoint**
+
+The REST endpoint may receive connections from external processes, including tools that are not part
+of Flink (for example curl request to the REST API). Setting up a proper certificate that is signed
+though a CA hierarchy may make sense for the REST endpoint.
+
+However, as mentioned above, the REST endpoint does not authenticate clients and thus typically needs
+to be secured via a proxy anyways.
+
+**REST Endpoint (simple self signed certificate)**
+
+This example shows how to create a simple keystore / truststore pair. The truststore does not contain
+the primary key and can be shared with other applications. In this example, *myhost.company.org / ip:10.0.2.15*
+is the node (or service) for the JobManager.
+
+```bash
+$ keytool -genkeypair -alias flink.rest -keystore rest.keystore -dname "CN=myhost.company.org" -ext "SAN=dns:myhost.company.org,ip:10.0.2.15" -storepass rest_keystore_password -keyalg RSA -keysize 4096 -storetype PKCS12
Review comment:
Some of the keytool commands are broken with `\` some of not. It would be good to make it consistent because it's hard to read it.
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,231 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This document briefly describes how Flink security works in the context of various deployment
+mechanisms (Standalone, native Kubernetes, YARN), filesystems, connectors, and state backends.
+
+## Deployment Modes
+
+Here is some information specific to each deployment mode.
+
+### Standalone Mode
+
+Steps to run a secure Flink cluster in standalone/cluster mode:
+
+1. Add security-related configuration options to the Flink configuration file (on all cluster nodes)
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path indicated by `security.kerberos.login.keytab` on
+ all cluster nodes.
+3. Deploy Flink cluster as normal.
+
+### Native Kubernetes and YARN Mode
Review comment:
Not sure where it belongs but there is a env var which has major effect on tokens and I don't see it anywhere. It would be good to mention it: `HADOOP_TOKEN_FILE_LOCATION`.
##########
File path: docs/content/docs/deployment/security/updates.md
##########
@@ -0,0 +1,73 @@
+---
Review comment:
Here I have an offtopic question just for my own understanding. @zentol, how do we handle security fixes in general? A public list of fixed security issues help users to double check what is fixed or not but this is true for hackers as well. This is the reason why many projects are not creating security jiras + fixes are added in a hidden way sometimes in totally unrelated PRs.
##########
File path: docs/content/docs/deployment/security/ssl.md
##########
@@ -0,0 +1,243 @@
+---
+title: "Encryption and Authentication using SSL"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster](#).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+Most of these connections are exposed via REST/HTTP endpoints (and used by the web UI). Some external
+services used as sources or sinks may use some other network protocol.
+
+The server will, by default, accept connections from any client, meaning that the REST endpoint does
+not authenticate the client. These REST endpoints, however, can be configured to require SSL encryption
+and mutual authentication.
+
+However, the recommended approach is setting up and configuring a dedicated proxy service (a "sidecar
+proxy") that controls access to the REST endpoint. This involves binding the REST endpoint to the
+loopback interface (or the pod-local interface in Kubernetes) and starting a REST proxy that authenticates
+and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/)
+or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html).
+
+The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety
+of authentication options and thus better integration into existing infrastructures.
+
+## Queryable State
+
+Connections to the [queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}})
+endpoints is currently not authenticated or encrypted.
+
+## SSL Setups
+
+{{< img src="/fig/ssl_mutual_auth.svg" alt="SSL Mutual Authentication" width=75% >}}
Review comment:
Personally I don't see the image, maybe local issue?!
<img width="427" alt="Screenshot 2022-02-16 at 12 51 40" src="https://user-images.githubusercontent.com/18561820/154259238-2e7e4f58-5a1a-4083-9e7e-a3590ce8a088.png">
##########
File path: docs/content/docs/deployment/security/overview.md
##########
@@ -0,0 +1,68 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL](https://www.ssl.com/faqs/faq-what-is-ssl/).
+
+When we talk about security for Flink, we generally make a distinction between securing the internal
+communication within the Flink cluster (i.e. between the Task Managers, between the Task Managers and
+the Flink Master) and securing the external communication between the cluster and the outside world.
+
+Internally, netty is used for the TCP connections used for data exchange among the task managers,
+and Akka is used for RPC between the Flink master and the task managers.
+
+Externally, HTTP is used for pretty much everything, except that some external services used as sources
+or sinks may use some other network protocol.
+
+## What is supported?
+
+Security enhancement features by the Flink community make it easy to access secured data, protect
+associated credentials, and increase overall security in a Flink cluster. The following security
+measures are currently supported:
+
+- Authentication of connections between Flink processes
+- Encryption of data transferred between Flink processes using SSL (Note that there is a performance
+ degradation when SSL is enabled, the magnitude of which depends on the CPU type and the JVM implementation.)
Review comment:
And heavily influenced by the key size.
##########
File path: docs/content/docs/deployment/security/overview.md
##########
@@ -0,0 +1,68 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL](https://www.ssl.com/faqs/faq-what-is-ssl/).
Review comment:
I know a normal user doesn't understand the difference between SSL and TLS. We use SSL in docs and configs for historical reasons (which is fine) but in the background do we enforce TLS? I'm asking it because SSL v3.0 deemed insecure in 2004 due to the POODLE attack.
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,231 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This document briefly describes how Flink security works in the context of various deployment
+mechanisms (Standalone, native Kubernetes, YARN), filesystems, connectors, and state backends.
+
+## Deployment Modes
+
+Here is some information specific to each deployment mode.
+
+### Standalone Mode
+
+Steps to run a secure Flink cluster in standalone/cluster mode:
+
+1. Add security-related configuration options to the Flink configuration file (on all cluster nodes)
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path indicated by `security.kerberos.login.keytab` on
+ all cluster nodes.
+3. Deploy Flink cluster as normal.
+
+### Native Kubernetes and YARN Mode
+
+Steps to run a secure Flink cluster in native Kubernetes and YARN mode:
+
+1. Add security-related configuration options to the Flink configuration file on the client
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path as indicated by `security.kerberos.login.keytab` on
+ the client node.
+3. Deploy Flink cluster as normal.
+
+In YARN and native Kubernetes mode, the keytab is automatically copied from the client to the Flink
+containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file can be
+either fetched from the cluster environment or uploaded by Flink. In the latter case, you need to
+configure the `security.kerberos.krb5-conf.path` to indicate the path of the Kerberos configuration
+file and Flink will copy this file to its containers/pods.
+
+For more information, see the [documentation on YARN security](https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md).
+
+#### Using `kinit` (YARN only)
+
+In YARN mode, it is possible to deploy a secure Flink cluster without a keytab, using only the ticket
+cache (as managed by `kinit`). This avoids the complexity of generating a keytab and avoids entrusting
+the cluster manager with it. In this scenario, the Flink CLI acquires Hadoop delegation tokens (for
+HDFS and for HBase). The main drawback is that the cluster is necessarily short-lived since the generated
+delegation tokens will expire (typically within a week).
+
+Steps to run a secure Flink cluster using `kinit`:
+
+1. Add security-related configuration options to the Flink configuration file on the client
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Login using the `kinit` command.
+3. Deploy Flink cluster as normal.
+
+
+## SSL - Tips for YARN Deployment
+
+For YARN, you can use the tools of Yarn to help:
+
+- Configuring security for internal communication is exactly the same as in the example above.
+
+- To secure the REST endpoint, you need to issue the REST endpoint's certificate such that it is
+ valid for all hosts that the JobManager may get deployed to. This can be done with a wild card
+ DNS name, or by adding multiple DNS names.
+
+- The easiest way to deploy keystores and truststore is by YARN client's *ship files* option (`-yt`).
+ Copy the keystore and truststore files into a local directory (say `deploy-keys/`) and start the
+ YARN session as follows: `flink run -m yarn-cluster -yt deploy-keys/ flinkapp.jar`
+
+- When deployed using YARN, Flink's web dashboard is accessible through YARN proxy's Tracking URL.
+ To ensure that the YARN proxy is able to access Flink's HTTPS URL, you need to configure YARN proxy
+ to accept Flink's SSL certificates.
+ For that, add the custom CA certificate into Java's default truststore on the YARN Proxy node.
+
+
+## Creating and Deploying Keystores and Truststores
+
+Keys, Certificates, and the Keystores and Truststores can be generated using the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html).
+You need to have an appropriate Java Keystore and Truststore accessible from each node in the Flink cluster.
+
+- For standalone setups, this means copying the files to each node, or adding them to a shared mounted directory.
+- For container based setups, add the keystore and truststore files to the container images.
+- For Yarn setups, the cluster deployment phase can automatically distribute the keystore and truststore files.
+
+For the externally facing REST endpoint, the common name or subject alternative names in the certificate
+should match the node's hostname and IP address.
+
+## Example SSL Setup Standalone and Kubernetes
+
+**Internal Connectivity**
+
+Execute the following keytool commands to create a key pair in a keystore:
+
+```bash
+$ keytool -genkeypair \
+ -alias flink.internal \
+ -keystore internal.keystore \
+ -dname "CN=flink.internal" \
+ -storepass internal_store_password \
+ -keyalg RSA \
+ -keysize 4096 \
+ -storetype PKCS12
+```
+
+The single key/certificate in the keystore is used the same way by the server and client endpoints
+(mutual authentication). The key pair acts as the shared secret for internal security, and we can
+directly use it as keystore and truststore.
+
+```yaml
+security.ssl.internal.enabled: true
+security.ssl.internal.keystore: /path/to/flink/conf/internal.keystore
+security.ssl.internal.truststore: /path/to/flink/conf/internal.keystore
+security.ssl.internal.keystore-password: internal_store_password
+security.ssl.internal.truststore-password: internal_store_password
+security.ssl.internal.key-password: internal_store_password
+```
+
+**REST Endpoint**
+
+The REST endpoint may receive connections from external processes, including tools that are not part
+of Flink (for example curl request to the REST API). Setting up a proper certificate that is signed
+though a CA hierarchy may make sense for the REST endpoint.
+
+However, as mentioned above, the REST endpoint does not authenticate clients and thus typically needs
+to be secured via a proxy anyways.
+
+**REST Endpoint (simple self signed certificate)**
+
+This example shows how to create a simple keystore / truststore pair. The truststore does not contain
+the primary key and can be shared with other applications. In this example, *myhost.company.org / ip:10.0.2.15*
Review comment:
You mean private key instead of primary, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] joemoe commented on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
joemoe commented on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1044537319
While this PR might lift the existing documentation to a sane state I think we should expand the security section. As it only says _fix security_ in the purpose of this PR I'm not sure if this is/should be the scope.
I'd like to guide users from their starting points towards the implemented features that could solve security concerns for them. Answering questions like:
* What is the surface of Apache Flink?
* Where are its flaws?
* How should they be addressed?
* What does Apache Flink offer?
* What do users need to solve outside of Apache Flink?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r809116052
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
Review comment:
Because that is what it originally said in the docs. Is this not true?
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/security/security-kerberos/#how-flink-security-works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r809117734
##########
File path: docs/content/docs/deployment/security/ssl.md
##########
@@ -0,0 +1,243 @@
+---
+title: "Encryption and Authentication using SSL"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster](#).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+Most of these connections are exposed via REST/HTTP endpoints (and used by the web UI). Some external
Review comment:
I am not certain. I saw this off some Flink training slides. So ALL of these external connections do use these endpoints?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r838422656
##########
File path: docs/content/docs/security/overview.md
##########
@@ -0,0 +1,87 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL/TLS](https://www.ssl.com/faqs/faq-what-is-ssl/).
+
+{{< hint info >}}
+SSL and TLS are both encryption protocols for the transport layer, used for encrypting data between
+the client and server. The TLS protocol is the successor to SSL. Flink enforces TLS by default, but
+you can also switch back to SSL.
+{{< /hint >}}
+
+## What needs to be secured in Flink?
+
+When we talk about security for Flink, we generally make a distinction between securing the internal
+communication within the Flink cluster (i.e. between the TaskManagers, between the TaskManagers and
+the JobManager) and securing the external communication between the cluster and the outside world.
+
+Internally, [Netty](https://netty.io) is used for the TCP connections used for data exchange among
+the TaskManagers, and [Akka](https://akka.io) is used for RPC between the JobManager and the TaskManagers.
+
+Externally, HTTP is used for pretty much everything, except that some external services used as sources
+or sinks may use some other network protocol.
+
+## What is supported by Flink?
+
+Security enhancement features by the Flink community make it easy to access secured data, protect
+associated credentials, and increase overall security in a Flink cluster. The following security
+measures are currently supported:
+
+- Authentication of connections between Flink processes
+- Encryption of data transferred between Flink processes using SSL (Note that there is a performance
+ degradation when SSL is enabled, the magnitude of which depends on the CPU type, the JVM implementation,
+ and the key size.)
+- Authorization of read / write operations by clients
+- Authorization is pluggable and integration with external authorization services is supported
+
+It is worth noting that security needs to be manually configured because the overall philosophy in
+Flink is to have defaults that work out-of-the-box, without any configuration. Thus, non-secured
+clusters are supported, as well as a mix of authenticated, unauthenticated, encrypted and unencrypted
+clients.
+
+## Where are its flaws? How should they be addressed?
+
+Web UI unsecure
Review comment:
These 2 statement lines are a little bit oddly formed. If I would a be a user not sure I would get what does it actually mean.
##########
File path: docs/content/docs/security/overview.md
##########
@@ -0,0 +1,87 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL/TLS](https://www.ssl.com/faqs/faq-what-is-ssl/).
+
+{{< hint info >}}
+SSL and TLS are both encryption protocols for the transport layer, used for encrypting data between
+the client and server. The TLS protocol is the successor to SSL. Flink enforces TLS by default, but
+you can also switch back to SSL.
+{{< /hint >}}
+
+## What needs to be secured in Flink?
+
+When we talk about security for Flink, we generally make a distinction between securing the internal
+communication within the Flink cluster (i.e. between the TaskManagers, between the TaskManagers and
+the JobManager) and securing the external communication between the cluster and the outside world.
+
+Internally, [Netty](https://netty.io) is used for the TCP connections used for data exchange among
+the TaskManagers, and [Akka](https://akka.io) is used for RPC between the JobManager and the TaskManagers.
+
+Externally, HTTP is used for pretty much everything, except that some external services used as sources
+or sinks may use some other network protocol.
+
+## What is supported by Flink?
+
+Security enhancement features by the Flink community make it easy to access secured data, protect
+associated credentials, and increase overall security in a Flink cluster. The following security
+measures are currently supported:
+
+- Authentication of connections between Flink processes
+- Encryption of data transferred between Flink processes using SSL (Note that there is a performance
+ degradation when SSL is enabled, the magnitude of which depends on the CPU type, the JVM implementation,
+ and the key size.)
+- Authorization of read / write operations by clients
+- Authorization is pluggable and integration with external authorization services is supported
+
+It is worth noting that security needs to be manually configured because the overall philosophy in
+Flink is to have defaults that work out-of-the-box, without any configuration. Thus, non-secured
+clusters are supported, as well as a mix of authenticated, unauthenticated, encrypted and unencrypted
+clients.
+
+## Where are its flaws? How should they be addressed?
+
+Web UI unsecure
+not secure by default
+
+## What needs to be solved outside of Flink?
+
+connectors
Review comment:
Nit: Shouldn't this be `Connectors` to follow the same pattern which we have in other chapters (capital first letter)?
##########
File path: docs/content/docs/security/running-cluster.md
##########
@@ -0,0 +1,193 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This guide describes how Flink security works in the context of various [deployment modes]({{< ref "docs/deployment/overview" >}}),
+[file systems]({{< ref "docs/deployment/filesystems/overview" >}}), [connectors]({{< ref "docs/connectors" >}}),
+and [state backends]({{< ref "docs/ops/state/state_backends" >}}).
+
+## Running a Secure Flink Cluster in Standalone Mode
+
+Standalone mode is the most barebone way of deploying a Flink cluster where Flink processes are launched
+as processes on the operating system. You can learn more [here]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on all cluster nodes.
+2. Ensure that the Kerberos keytab file exists at the path indicated by the `security.kerberos.login.keytab`
+ setting on all cluster nodes.
+3. Deploy the Flink cluster as normal.
+
+{{< hint info >}}
+`HADOOP_TOKEN_FILE_LOCATION` is an important environment variable that specifies the location of your
+credential files/tokens. The service instances uses this location to find the file to load the credentials
+and run the tasks. Launched containers must load the delegation tokens from this location, and use them
+(including renewals) until they can no longer be renewed.
+{{< /hint >}}
+
+You need to have an appropriate Java keystore and truststore accessible from each node in the Flink
+cluster. For standalone setups, this means copying the files to each node, or adding them to a shared
+mounted directory. For externally facing REST endpoints, the common name or subject alternative names
+in the certificate should match the node's hostname and IP address.
+
+Keystores and truststores can be generated using the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html).
+
+## Running a Secure Flink Cluster in Kubernetes
+
+Flink’s native Kubernetes integration allows you to directly deploy Flink on a running Kubernetes cluster.
+You can learn more [here]({{< ref "docs/deployment/resource-providers/native_kubernetes" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on the client node.
+2. Ensure that the Kerberos keytab file exists at the path as indicated by the `security.kerberos.login.keytab`
+ on the client node.
+3. Deploy the Flink cluster as normal.
+
+### Setting up Kerberos
+
+When deploying Flink on Kubernetes, the Kerberos keytab file is automatically copied from the client
+node to the Flink containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file should
+be present in the cluster nodes / classpath. If you want Flink to upload a new file, you need to configure
+the `security.kerberos.krb5-conf.path` setting to indicate the path of the Kerberos configuration file
+and Flink will copy this file to its containers/pods.
+
+{{< hint info >}}
+`HADOOP_TOKEN_FILE_LOCATION` is an important environment variable that specifies the location of your
+credential files/tokens. The service instances uses this location to find the file to load the credentials
+and run the tasks. Launched containers must load the delegation tokens from this location, and use them
+(including renewals) until they can no longer be renewed.
+{{< /hint >}}
+
+### Setting up keystores and truststores
+
+You need to have an appropriate Java keystore and truststore accessible from each node in the Flink
+cluster. For container-based setups, this means adding the keystore and truststore files to the container
+images. For externally facing REST endpoints, the common name or subject alternative names in the
+certificate should match the node's hostname and IP address.
+
+## Running a Secure Flink Cluster in YARN
+
+Flink services are submitted to YARN’s ResourceManager, which spawns containers on machines managed
+by YARN NodeManagers. Flink deploys its JobManager and TaskManager instances into such containers.
+You can learn more [here]({{< ref "docs/deployment/resource-providers/yarn" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on the client node.
+2. Ensure that the Kerberos keytab file exists at the path as indicated by the `security.kerberos.login.keytab`
+ on the client node.
+3. Deploy the Flink cluster as normal.
+
+### Setting up Kerberos
+
+When deploying Flink on YARN, the Kerberos keytab is automatically copied from the client node to the
+Flink containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file can be
Review comment:
This is not yet answered what does that mean: `This file can be either fetched from the cluster environment`
##########
File path: docs/content/docs/security/ssl.md
##########
@@ -0,0 +1,367 @@
+---
+title: "Encryption and Authentication using SSL/TLS"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL/TLS for Cluster Components
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster]({{< ref "docs/security/running-cluster" >}}).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+These connections are exposed via REST/HTTP endpoints (and are used by Flink's web UI). Some
+application-specific external services such as sources and sinks may use some other network protocol.
+
+The server will, by default, accept connections from any client, meaning that the REST endpoint does
+not authenticate the client. These REST endpoints, however, can be configured to require SSL encryption
+and mutual authentication.
+
+However, the recommended approach is setting up and configuring a dedicated proxy service (a "sidecar
+proxy") that controls access to the REST endpoint. This involves binding the REST endpoint to the
+loopback interface (or the pod-local interface in Kubernetes) and starting a REST proxy that authenticates
+and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/)
+or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html).
+
+The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety
+of authentication options and thus better integration into existing infrastructures.
+
+## Queryable State
+
+Connections to the [queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}})
+endpoints is currently not authenticated or encrypted.
+
+## SSL Setups
+
+{{< img src="/fig/ssl_mutual_auth.svg" alt="SSL Mutual Authentication" width=75% >}}
Review comment:
This is still not working. Not sure what is the issue but having such issue is something we need to fix I think.
<img width="427" alt="Screenshot 2022-02-16 at 12 51 40" src="https://user-images.githubusercontent.com/18561820/154259238-2e7e4f58-5a1a-4083-9e7e-a3590ce8a088.png">
##########
File path: docs/content/docs/security/ssl.md
##########
@@ -0,0 +1,367 @@
+---
+title: "Encryption and Authentication using SSL/TLS"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL/TLS for Cluster Components
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster]({{< ref "docs/security/running-cluster" >}}).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+These connections are exposed via REST/HTTP endpoints (and are used by Flink's web UI). Some
+application-specific external services such as sources and sinks may use some other network protocol.
+
+The server will, by default, accept connections from any client, meaning that the REST endpoint does
+not authenticate the client. These REST endpoints, however, can be configured to require SSL encryption
+and mutual authentication.
+
+However, the recommended approach is setting up and configuring a dedicated proxy service (a "sidecar
+proxy") that controls access to the REST endpoint. This involves binding the REST endpoint to the
+loopback interface (or the pod-local interface in Kubernetes) and starting a REST proxy that authenticates
+and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/)
+or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html).
+
+The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety
+of authentication options and thus better integration into existing infrastructures.
+
+## Queryable State
+
+Connections to the [queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}})
+endpoints is currently not authenticated or encrypted.
+
+## SSL Setups
+
+{{< img src="/fig/ssl_mutual_auth.svg" alt="SSL Mutual Authentication" width=75% >}}
+
+Each participant has a keystore and a truststore, which are files.
+
+A keystore contains a certificate (which contains a public key) and a private key. A truststore
+contains trusted certificates and certificate chains/authorities.
+
+Establishing encrypted, authenticated communication is a multi-step process, shown in the figure.
+Certificates are exchanged and validated against the truststore, after which the two parties can
+safely communicate.
+
+### Typical SSL Setup in Flink
+
+For mutually authenticated internal connections, note that:
+
+- a keystore and a truststore can contain the same dedicated certificate
+- the same file can be used for both keystore and truststore
+- wildcard hostnames or addresses can be used
+
+For internal communication between servers in a Flink cluster, a secure setup can be established with
+a single, self-signed certificate that all parties use as both their keystore and truststore. You can
+also use this approach for external communication when establishing mutual authentication for communication
+between clients and the Flink Master.
+
+### Configuring Keystores and Truststores
+
+The SSL configuration requires configuring a keystore and a truststore such that the truststore trusts
+the keystore's certificate.
+
+You can use the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html)
+to generate keys, certificates, keystores, and truststores:
+
+```bash
+ keytool -genkeypair -alias flink.internal -keystore internal.keystore \
Review comment:
Is there a specific reason why this keytool command is broken with `\` and all the others not?
##########
File path: docs/content/docs/security/running-cluster.md
##########
@@ -0,0 +1,193 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This guide describes how Flink security works in the context of various [deployment modes]({{< ref "docs/deployment/overview" >}}),
+[file systems]({{< ref "docs/deployment/filesystems/overview" >}}), [connectors]({{< ref "docs/connectors" >}}),
+and [state backends]({{< ref "docs/ops/state/state_backends" >}}).
+
+## Running a Secure Flink Cluster in Standalone Mode
+
+Standalone mode is the most barebone way of deploying a Flink cluster where Flink processes are launched
+as processes on the operating system. You can learn more [here]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on all cluster nodes.
+2. Ensure that the Kerberos keytab file exists at the path indicated by the `security.kerberos.login.keytab`
+ setting on all cluster nodes.
+3. Deploy the Flink cluster as normal.
+
+{{< hint info >}}
+`HADOOP_TOKEN_FILE_LOCATION` is an important environment variable that specifies the location of your
+credential files/tokens. The service instances uses this location to find the file to load the credentials
+and run the tasks. Launched containers must load the delegation tokens from this location, and use them
+(including renewals) until they can no longer be renewed.
+{{< /hint >}}
+
+You need to have an appropriate Java keystore and truststore accessible from each node in the Flink
+cluster. For standalone setups, this means copying the files to each node, or adding them to a shared
+mounted directory. For externally facing REST endpoints, the common name or subject alternative names
+in the certificate should match the node's hostname and IP address.
+
+Keystores and truststores can be generated using the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html).
+
+## Running a Secure Flink Cluster in Kubernetes
+
+Flink’s native Kubernetes integration allows you to directly deploy Flink on a running Kubernetes cluster.
+You can learn more [here]({{< ref "docs/deployment/resource-providers/native_kubernetes" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on the client node.
+2. Ensure that the Kerberos keytab file exists at the path as indicated by the `security.kerberos.login.keytab`
+ on the client node.
+3. Deploy the Flink cluster as normal.
+
+### Setting up Kerberos
+
+When deploying Flink on Kubernetes, the Kerberos keytab file is automatically copied from the client
+node to the Flink containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file should
+be present in the cluster nodes / classpath. If you want Flink to upload a new file, you need to configure
+the `security.kerberos.krb5-conf.path` setting to indicate the path of the Kerberos configuration file
+and Flink will copy this file to its containers/pods.
+
+{{< hint info >}}
+`HADOOP_TOKEN_FILE_LOCATION` is an important environment variable that specifies the location of your
+credential files/tokens. The service instances uses this location to find the file to load the credentials
+and run the tasks. Launched containers must load the delegation tokens from this location, and use them
Review comment:
`Launched containers can load the delegation tokens from this location` because Flink can fetch it in the initial phase too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1048563745
@infoverload
"This branch has conflicts that must be resolved"
This can be resolved by merging the latest master into this branch. That case (by not doing a rebase but a merge) the discussion thread here remains.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] knaufk commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
knaufk commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r808778983
##########
File path: docs/content/docs/deployment/security/overview.md
##########
@@ -0,0 +1,68 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL](https://www.ssl.com/faqs/faq-what-is-ssl/).
+
+When we talk about security for Flink, we generally make a distinction between securing the internal
+communication within the Flink cluster (i.e. between the Task Managers, between the Task Managers and
+the Flink Master) and securing the external communication between the cluster and the outside world.
Review comment:
We don't use the term "Flink Master" anymore. It is just called "Jobmanager".
##########
File path: docs/content/docs/deployment/security/overview.md
##########
@@ -0,0 +1,68 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
Review comment:
I think, Security should be a top-level section sibling of "Deployment".
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
Review comment:
Flink only has first-class support for Kerberos? Why do you think so?
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,292 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This guide describes how Flink security works in the context of various [deployment modes]({{< ref "docs/deployment/overview" >}}),
Review comment:
This page seems to focus on different resource providers not deployment modes, right?
##########
File path: docs/content/docs/deployment/security/overview.md
##########
@@ -0,0 +1,68 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL](https://www.ssl.com/faqs/faq-what-is-ssl/).
+
+When we talk about security for Flink, we generally make a distinction between securing the internal
+communication within the Flink cluster (i.e. between the Task Managers, between the Task Managers and
+the Flink Master) and securing the external communication between the cluster and the outside world.
+
+Internally, netty is used for the TCP connections used for data exchange among the task managers,
+and Akka is used for RPC between the Flink master and the task managers.
+
+Externally, HTTP is used for pretty much everything, except that some external services used as sources
+or sinks may use some other network protocol.
+
+## What is supported?
+
+Security enhancement features by the Flink community make it easy to access secured data, protect
+associated credentials, and increase overall security in a Flink cluster. The following security
+measures are currently supported:
+
+- Authentication of connections between Flink processes
+- Encryption of data transferred between Flink processes using SSL (Note that there is a performance
+ degradation when SSL is enabled, the magnitude of which depends on the CPU type and the JVM implementation.)
+- Authorization of read / write operations by clients
+- Authorization is pluggable and integration with external authorization services is supported
+
+It is worth noting that security is optional because the overall philosophy in Flink is to have defaults
Review comment:
"Security is optional" is our "philosophy" does not sound good. ;)
##########
File path: docs/content/docs/deployment/security/overview.md
##########
@@ -0,0 +1,68 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL](https://www.ssl.com/faqs/faq-what-is-ssl/).
+
+When we talk about security for Flink, we generally make a distinction between securing the internal
+communication within the Flink cluster (i.e. between the Task Managers, between the Task Managers and
Review comment:
Taskmanager is usually one word.
##########
File path: docs/content/docs/deployment/security/ssl.md
##########
@@ -0,0 +1,243 @@
+---
+title: "Encryption and Authentication using SSL"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster](#).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+Most of these connections are exposed via REST/HTTP endpoints (and used by the web UI). Some external
Review comment:
Are there any connections that don't use these endpoints?
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,292 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This guide describes how Flink security works in the context of various [deployment modes]({{< ref "docs/deployment/overview" >}}),
Review comment:
I am wondering if the the resource provider specifics should actually go with the resource providers and we only reference there and back. :thinking:
##########
File path: docs/content/docs/deployment/security/ssl.md
##########
@@ -0,0 +1,243 @@
+---
+title: "Encryption and Authentication using SSL"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster](#).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+Most of these connections are exposed via REST/HTTP endpoints (and used by the web UI). Some external
+services used as sources or sinks may use some other network protocol.
+
+The server will, by default, accept connections from any client, meaning that the REST endpoint does
+not authenticate the client. These REST endpoints, however, can be configured to require SSL encryption
+and mutual authentication.
+
+However, the recommended approach is setting up and configuring a dedicated proxy service (a "sidecar
+proxy") that controls access to the REST endpoint. This involves binding the REST endpoint to the
+loopback interface (or the pod-local interface in Kubernetes) and starting a REST proxy that authenticates
+and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/)
+or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html).
+
+The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety
+of authentication options and thus better integration into existing infrastructures.
+
+## Queryable State
+
+Connections to the [queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}})
+endpoints is currently not authenticated or encrypted.
+
+## SSL Setups
+
+{{< img src="/fig/ssl_mutual_auth.svg" alt="SSL Mutual Authentication" width=75% >}}
+
+Each participant has a keystore and a truststore, which are files.
+
+A keystore contains a certificate (which contains a public key) and a private key. A truststore
+contains trusted certificates and certificate chains/authorities.
+
+Establishing encrypted, authenticated communication is a multi-step process, shown in the figure.
+Certificates are exchanged and validated against the truststore, after which the two parties can
+safely communicate.
+
+### Typical SSL Setup in Flink
+
+For mutually authenticated internal connections, note that:
+
+- a keystore and a truststore can contain the same dedicated certificate
+- the same file can be used for both keystore and truststore
+- wildcard hostnames or addresses can be used
+
+For internal communication between servers in a Flink cluster, a secure setup can be established with
+a single, self-signed certificate that all parties use as both their keystore and truststore. You can
+also use this approach for external communication when establishing mutual authentication for communication
+between clients and the Flink Master.
+
+### Configuring Keystores and Truststores
+
+The SSL configuration requires configuring a keystore and a truststore such that the truststore trusts
+the keystore's certificate.
+
+You can use the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html)
+to generate keys, certificates, keystores, and truststores:
+
+```bash
+ keytool -genkeypair -alias flink.internal -keystore internal.keystore \
+ -dname "CN=flink.internal" -storepass internal_store_password -keyalg RSA \
+ -keysize 4096 -storetype PKCS12
+```
+
+| Deployment mode | How to add the files |
+|------------------------|-------------------------------------------------------------------------|
+| Standalone clusters | copy the files to each node, or add them to a shared mounted filesystem |
+| Containerized clusters | add the files to the container images |
+| YARN | the cluster deployment phase can distribute these files |
+
+### Using Cipher Suites
+
+While the acts of encryption and decryption themselves are performed by keys, cipher suites outline
+the set of steps that the keys must follow to do so and the order in which these steps are executed.
+There are numerous cipher suites out there, each one with varying instructions on the encryption and
+decryption process.
+
+{{< hint warning >}}
+The [IETF RFC 7525](https://tools.ietf.org/html/rfc7525) recommends using a specific set of cipher
+suites for strong security. Since these cipher suites are not available on many setups out-of-the-box,
+Flink defaults to TLS_RSA_WITH_AES_128_CBC_SHA (a slightly weaker but more widely available cipher suite).
+
+If stronger encryption is available in your environment, we recommend that you update your SSL setup
+to the stronger cipher suites by adding the below entry to the Flink configuration file (`flink-conf.yaml`):
+
+```yaml
+security.ssl.algorithms: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
+```
+
+If these cipher suites are not supported in your setup, you will see that Flink processes will not
+be able to connect to each other.
+{{< /hint >}}
+
+### Configuring SSL for Internal Connectivity
+
+The following setting in `flink-conf.yaml` is used to enable SSL for all internal connections:
+
+```yaml
+security.ssl.internal.enabled: true
+```
+
+{{< hint info >}}
+For backwards compatibility, the **security.ssl.enabled** option still exists and enables SSL
+for both internal and external/REST endpoints.
+{{< /hint >}}
+
+You can disable security for different connection types. When `security.ssl.internal.enabled` is set
+to `true`, you can set the following parameters to `false` to disable SSL for that particular connection
+type:
+
+- `taskmanager.data.ssl.enabled` → Data communication between TaskManagers
+- `blob.service.ssl.enabled` → Transport of BLOBs from JobManager to TaskManager
+- `akka.ssl.enabled` → Akka-based RPC connections between JobManager / TaskManager / ResourceManager
+
+Because internal communication is mutually authenticated between the server and the client, keystore
+and truststore typically refer to a dedicated certificate that acts as a shared secret. In such a setup,
+the certificate can use wildcard hostnames or addresses. When using self-signed certificates, it is
+even possible to use the same file as keystore and truststore.
+
+Take note of the following configuration settings:
+
+```yaml
+security.ssl.internal.keystore: /path/to/file.keystore
+security.ssl.internal.keystore-password: keystore_password
+security.ssl.internal.key-password: key_password
+security.ssl.internal.truststore: /path/to/file.truststore
+security.ssl.internal.truststore-password: truststore_password
+```
+
+When using a certificate that is not self-signed, but signed by Certified Authorities (CA), you need
+to use certificate pinning to allow only a specific certificate to be trusted when establishing the
+connectivity:
+
+```yaml
+security.ssl.internal.cert.fingerprint: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
+```
+
+### Configuring SSL for External Connectivity (REST Endpoints)
+
+The following setting in `flink-conf.yaml` is used to enable SSL for REST/external connections:
+
+```yaml
+security.ssl.rest.enabled: true
+```
+
+{{< hint info >}}
+For backwards compatibility, the **security.ssl.enabled** option still exists and enables SSL
+for both internal and external/REST endpoints.
+{{< /hint >}}
+
+By default, the keystore is used by the server REST endpoints, and the truststore is used
+by the REST clients (including the CLI client) to accept the server's certificate. In the case where
+the REST keystore has a self-signed certificate, the truststore must trust that certificate directly.
+If the REST endpoint uses a certificate that is signed through a proper certification hierarchy, the
+roots of that hierarchy should be in the truststore.
+
+If mutual authentication is enabled, the keystore and the truststore are used by both the server
+endpoints and the REST clients.
+
+Take note of the following configuration settings:
+
+```yaml
+security.ssl.rest.keystore: /path/to/file.keystore
+security.ssl.rest.keystore-password: keystore_password
+security.ssl.rest.key-password: key_password
+security.ssl.rest.truststore: /path/to/file.truststore
+security.ssl.rest.truststore-password: truststore_password
+security.ssl.rest.authentication-enabled: false
+```
+
+### Complete List of SSL Options
+
+{{< generated/security_configuration >}}
Review comment:
These configuration options not only include SSL but also Kerberos/SASL related options.
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,231 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This document briefly describes how Flink security works in the context of various deployment
+mechanisms (Standalone, native Kubernetes, YARN), filesystems, connectors, and state backends.
+
+## Deployment Modes
+
+Here is some information specific to each deployment mode.
+
+### Standalone Mode
+
+Steps to run a secure Flink cluster in standalone/cluster mode:
+
+1. Add security-related configuration options to the Flink configuration file (on all cluster nodes)
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path indicated by `security.kerberos.login.keytab` on
+ all cluster nodes.
+3. Deploy Flink cluster as normal.
+
+### Native Kubernetes and YARN Mode
+
+Steps to run a secure Flink cluster in native Kubernetes and YARN mode:
+
+1. Add security-related configuration options to the Flink configuration file on the client
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Ensure that the keytab file exists at the path as indicated by `security.kerberos.login.keytab` on
+ the client node.
+3. Deploy Flink cluster as normal.
+
+In YARN and native Kubernetes mode, the keytab is automatically copied from the client to the Flink
+containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file can be
+either fetched from the cluster environment or uploaded by Flink. In the latter case, you need to
+configure the `security.kerberos.krb5-conf.path` to indicate the path of the Kerberos configuration
+file and Flink will copy this file to its containers/pods.
+
+For more information, see the [documentation on YARN security](https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md).
+
+#### Using `kinit` (YARN only)
+
+In YARN mode, it is possible to deploy a secure Flink cluster without a keytab, using only the ticket
+cache (as managed by `kinit`). This avoids the complexity of generating a keytab and avoids entrusting
+the cluster manager with it. In this scenario, the Flink CLI acquires Hadoop delegation tokens (for
+HDFS and for HBase). The main drawback is that the cluster is necessarily short-lived since the generated
+delegation tokens will expire (typically within a week).
+
+Steps to run a secure Flink cluster using `kinit`:
+
+1. Add security-related configuration options to the Flink configuration file on the client
+ (see [here]({{< ref "docs/deployment/config" >}}#auth-with-external-systems)).
+2. Login using the `kinit` command.
+3. Deploy Flink cluster as normal.
+
+
+## SSL - Tips for YARN Deployment
+
+For YARN, you can use the tools of Yarn to help:
+
+- Configuring security for internal communication is exactly the same as in the example above.
+
+- To secure the REST endpoint, you need to issue the REST endpoint's certificate such that it is
+ valid for all hosts that the JobManager may get deployed to. This can be done with a wild card
+ DNS name, or by adding multiple DNS names.
+
+- The easiest way to deploy keystores and truststore is by YARN client's *ship files* option (`-yt`).
+ Copy the keystore and truststore files into a local directory (say `deploy-keys/`) and start the
+ YARN session as follows: `flink run -m yarn-cluster -yt deploy-keys/ flinkapp.jar`
+
+- When deployed using YARN, Flink's web dashboard is accessible through YARN proxy's Tracking URL.
+ To ensure that the YARN proxy is able to access Flink's HTTPS URL, you need to configure YARN proxy
+ to accept Flink's SSL certificates.
+ For that, add the custom CA certificate into Java's default truststore on the YARN Proxy node.
+
+
+## Creating and Deploying Keystores and Truststores
+
+Keys, Certificates, and the Keystores and Truststores can be generated using the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html).
+You need to have an appropriate Java Keystore and Truststore accessible from each node in the Flink cluster.
+
+- For standalone setups, this means copying the files to each node, or adding them to a shared mounted directory.
+- For container based setups, add the keystore and truststore files to the container images.
+- For Yarn setups, the cluster deployment phase can automatically distribute the keystore and truststore files.
+
+For the externally facing REST endpoint, the common name or subject alternative names in the certificate
+should match the node's hostname and IP address.
+
+## Example SSL Setup Standalone and Kubernetes
+
+**Internal Connectivity**
+
+Execute the following keytool commands to create a key pair in a keystore:
+
+```bash
+$ keytool -genkeypair \
+ -alias flink.internal \
+ -keystore internal.keystore \
+ -dname "CN=flink.internal" \
+ -storepass internal_store_password \
+ -keyalg RSA \
+ -keysize 4096 \
+ -storetype PKCS12
+```
+
+The single key/certificate in the keystore is used the same way by the server and client endpoints
+(mutual authentication). The key pair acts as the shared secret for internal security, and we can
+directly use it as keystore and truststore.
+
+```yaml
+security.ssl.internal.enabled: true
+security.ssl.internal.keystore: /path/to/flink/conf/internal.keystore
+security.ssl.internal.truststore: /path/to/flink/conf/internal.keystore
+security.ssl.internal.keystore-password: internal_store_password
Review comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "112a4f6d047be628b5e0bc34ba54d246aea7f4c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "112a4f6d047be628b5e0bc34ba54d246aea7f4c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
* 112a4f6d047be628b5e0bc34ba54d246aea7f4c9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "112a4f6d047be628b5e0bc34ba54d246aea7f4c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "112a4f6d047be628b5e0bc34ba54d246aea7f4c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f510145df68812e0eed11126fa839e85dedd26f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055)
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
* 112a4f6d047be628b5e0bc34ba54d246aea7f4c9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d46bd54e8d4e05e791d1679c5305100ec614c4c1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542",
"triggerID" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"triggerType" : "PUSH"
}, {
"hash" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545",
"triggerID" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"triggerType" : "PUSH"
}, {
"hash" : "04b63296c8bd42f3191643ccc2990578c215e593",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "04b63296c8bd42f3191643ccc2990578c215e593",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec06532a2ea9edb00c82ce9304057966e5ef9ae7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545)
* 04b63296c8bd42f3191643ccc2990578c215e593 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r807907397
##########
File path: docs/content/docs/deployment/security/updates.md
##########
@@ -0,0 +1,73 @@
+---
+title: Security Updates
+weight: 6
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Updates
Review comment:
Same as https://github.com/apache/flink/pull/18746/files#r807906811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r807901125
##########
File path: docs/content/docs/deployment/security/updates.md
##########
@@ -0,0 +1,73 @@
+---
Review comment:
That's decided on a case-by-case basis.
Generally speaking we aim to publish vulnerabilities once a mitigation is in place (either a new release or patch, depending on the severity).
In the case of Log4j we communicated it right away because the cat was out of the bag already at that point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r808767412
##########
File path: docs/content/docs/deployment/security/ssl.md
##########
@@ -0,0 +1,243 @@
+---
+title: "Encryption and Authentication using SSL"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster](#).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+Most of these connections are exposed via REST/HTTP endpoints (and used by the web UI). Some external
+services used as sources or sinks may use some other network protocol.
+
+The server will, by default, accept connections from any client, meaning that the REST endpoint does
+not authenticate the client. These REST endpoints, however, can be configured to require SSL encryption
+and mutual authentication.
+
+However, the recommended approach is setting up and configuring a dedicated proxy service (a "sidecar
+proxy") that controls access to the REST endpoint. This involves binding the REST endpoint to the
+loopback interface (or the pod-local interface in Kubernetes) and starting a REST proxy that authenticates
+and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/)
+or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html).
+
+The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety
+of authentication options and thus better integration into existing infrastructures.
+
+## Queryable State
+
+Connections to the [queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}})
+endpoints is currently not authenticated or encrypted.
+
+## SSL Setups
+
+{{< img src="/fig/ssl_mutual_auth.svg" alt="SSL Mutual Authentication" width=75% >}}
Review comment:
I experienced this issue locally as well and I just could not figure out what is wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r810115995
##########
File path: docs/content/docs/deployment/security/kerberos.md
##########
@@ -0,0 +1,116 @@
+---
+title: Authentication with Kerberos
+weight: 2
+type: docs
+aliases:
+ - /deployment/security/kerberos.html
+ - /ops/security-kerberos.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Enabling and Configuring Authentication with Kerberos
+
+## What is Kerberos?
+
+[Kerberos](https://web.mit.edu/kerberos/) is a network authentication protocol that provides a secure,
+single-sign-on, trusted, third-party mutual authentication service. It is designed to provide strong
+authentication for client/server applications by using secret-key cryptography.
+
+## How the Flink Security Infrastructure works with Kerberos
+
+A Flink program may use first- or third-party connectors, necessitating arbitrary authentication methods
+(Kerberos, SSL/TLS, username/password, etc.). While satisfying the security requirements for all connectors
+is an ongoing effort, Flink provides first-class support for Kerberos authentication only.
+
+Kerberos can be used to authenticate connections to:
+- Hadoop and its components (YARN, HDFS, HBase)
+- ZooKeeper
+- Kafka (0.9+)
+
+The current implementation supports running Flink clusters (JobManager / TaskManager / Jobs) with two
+authentication modes:
+- a configured [Kerberos keytab](https://web.mit.edu/kerberos/krb5-devel/doc/basic/keytab_def.html) credential
+- [Hadoop delegation tokens](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html)
+
+In production deployments, streaming jobs usually run for long periods of time. It is important to be
+able to authenticate to secured data sources throughout the lifetime of the job. Kerberos keytabs are
+the preferred authentication approach because they won't expire during the lifetime of long-running
+stream processing applications, unlike a Hadoop delegation token or ticket cache entry.
+
+Note that the credentials are tied to a Flink cluster and not to a running job. Thus, all applications
+that run on the same cluster use the same authentication token and all jobs within a cluster will share
+the credentials configured for that cluster. If you need to work with different credentials, you should
+start a new cluster. For example, to use a different keytab for a certain job, simply launch a separate
+Flink cluster with a different configuration. Numerous Flink clusters may run side-by-side in a Kubernetes
+or YARN environment.
+
+Note that it is possible to enable and configure the use of Kerberos independently for each service
+or connector that is capable of being used with Kerberos. For example, you may enable Hadoop security
+without enabling the use of Kerberos for ZooKeeper, or vice versa.
+
+All services using Kerberos will use the same credentials. If you need to run some jobs with different
+Kerberos credentials, those jobs will have to run in a different cluster that is configured to use
+those other credentials. For example, you can decide to use Kerberos for Hadoop security, but not for ZooKeeper.
+
+## Using Kerberos with Flink Security Modules
+
+The internal architecture of Flink security is based on _security modules_ (which implements [`org.apache.flink.runtime.security.modules.SecurityModule`](https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/runtime/security/modules/SecurityModule.html)).
+These modules are installed at startup.
+
+### Hadoop Security Module
+
+This module uses the Hadoop `UserGroupInformation` (UGI) class to establish a process-wide *login user*
+context. The login user is then used for all interactions with Hadoop, including HDFS, HBase, and YARN.
+
+If Hadoop security is enabled (in `core-site.xml`), the login user will have whatever Kerberos credential
+is configured. Otherwise, the login user conveys only the user identity of the OS account that launched
+the cluster.
+
+### JAAS Security Module
+
+This module provides a dynamic [JAAS](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jaas/JAASRefGuide.html)
+configuration to the cluster, making available the configured Kerberos credential to ZooKeeper, Kafka,
+and other such components that rely on JAAS.
+
+Note that the user may also provide a static JAAS configuration file using the mechanisms described
+in the [Java SE Documentation](http://docs.oracle.com/javase/7/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html).
+Static entries override any dynamic entries provided by this module.
+
+### ZooKeeper Security Module
+
+This module configures certain process-wide ZooKeeper security-related settings, namely the ZooKeeper
+service name (default: `zookeeper`) and the JAAS login context name (default: `Client`).
+
+## Ticket Renewal
+
+A Ticket Granting Ticket (TGT) is a small, encrypted identification file with a limited validity period.
Review comment:
You've done it well, wiki is super useful most of the time but maintained by normal ppls like you and me. If I would go there and would state that TGT is bird which lives under the water then that would be the source of truth for some time. The statement is not completely bad but if one reads the official developer documentation then it's clear that wiki simplifies the truth. TGT MAY live in file but there are other possibilities, like API, DIR, FILE, KEYRING, etc...
If we stick to the MIT implementation then the closest documentation is the following: https://web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html
All in all wiki has good quality in general but in security tiny wording/meaning differences can change whole security strategies within companies. Normally when I read glitches in the security documentation of a product then I always have the feeling (which may of may not be true) that the guys who designed the secu model are not fully aware all the aspects of the system behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d46bd54e8d4e05e791d1679c5305100ec614c4c1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r812055107
##########
File path: docs/content/docs/deployment/security/ssl.md
##########
@@ -0,0 +1,243 @@
+---
+title: "Encryption and Authentication using SSL"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster](#).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+Most of these connections are exposed via REST/HTTP endpoints (and used by the web UI). Some external
+services used as sources or sinks may use some other network protocol.
+
+The server will, by default, accept connections from any client, meaning that the REST endpoint does
+not authenticate the client. These REST endpoints, however, can be configured to require SSL encryption
+and mutual authentication.
+
+However, the recommended approach is setting up and configuring a dedicated proxy service (a "sidecar
+proxy") that controls access to the REST endpoint. This involves binding the REST endpoint to the
+loopback interface (or the pod-local interface in Kubernetes) and starting a REST proxy that authenticates
+and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/)
+or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html).
+
+The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety
+of authentication options and thus better integration into existing infrastructures.
+
+## Queryable State
+
+Connections to the [queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}})
+endpoints is currently not authenticated or encrypted.
+
+## SSL Setups
+
+{{< img src="/fig/ssl_mutual_auth.svg" alt="SSL Mutual Authentication" width=75% >}}
+
+Each participant has a keystore and a truststore, which are files.
+
+A keystore contains a certificate (which contains a public key) and a private key. A truststore
+contains trusted certificates and certificate chains/authorities.
+
+Establishing encrypted, authenticated communication is a multi-step process, shown in the figure.
+Certificates are exchanged and validated against the truststore, after which the two parties can
+safely communicate.
+
+### Typical SSL Setup in Flink
+
+For mutually authenticated internal connections, note that:
+
+- a keystore and a truststore can contain the same dedicated certificate
+- the same file can be used for both keystore and truststore
+- wildcard hostnames or addresses can be used
+
+For internal communication between servers in a Flink cluster, a secure setup can be established with
+a single, self-signed certificate that all parties use as both their keystore and truststore. You can
+also use this approach for external communication when establishing mutual authentication for communication
+between clients and the Flink Master.
+
+### Configuring Keystores and Truststores
+
+The SSL configuration requires configuring a keystore and a truststore such that the truststore trusts
+the keystore's certificate.
+
+You can use the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html)
+to generate keys, certificates, keystores, and truststores:
+
+```bash
+ keytool -genkeypair -alias flink.internal -keystore internal.keystore \
+ -dname "CN=flink.internal" -storepass internal_store_password -keyalg RSA \
+ -keysize 4096 -storetype PKCS12
+```
+
+| Deployment mode | How to add the files |
+|------------------------|-------------------------------------------------------------------------|
+| Standalone clusters | copy the files to each node, or add them to a shared mounted filesystem |
+| Containerized clusters | add the files to the container images |
+| YARN | the cluster deployment phase can distribute these files |
+
+### Using Cipher Suites
+
+While the acts of encryption and decryption themselves are performed by keys, cipher suites outline
+the set of steps that the keys must follow to do so and the order in which these steps are executed.
+There are numerous cipher suites out there, each one with varying instructions on the encryption and
+decryption process.
+
+{{< hint warning >}}
+The [IETF RFC 7525](https://tools.ietf.org/html/rfc7525) recommends using a specific set of cipher
+suites for strong security. Since these cipher suites are not available on many setups out-of-the-box,
+Flink defaults to TLS_RSA_WITH_AES_128_CBC_SHA (a slightly weaker but more widely available cipher suite).
Review comment:
I agree. Perhaps you can start a discussion on the mailing list about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r819509463
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,292 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This guide describes how Flink security works in the context of various [deployment modes]({{< ref "docs/deployment/overview" >}}),
Review comment:
Or we could link/reference the other way around (resource provider specific pages to the security section)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
* bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542",
"triggerID" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
* bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
* bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d46bd54e8d4e05e791d1679c5305100ec614c4c1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386)
* 91c09fb42686497a7e80405effd8fe474c24d4df UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91c09fb42686497a7e80405effd8fe474c24d4df Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444)
* c4f76a841937d005508c859a643c6ac4c0fcbf7f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r807906811
##########
File path: docs/content/docs/deployment/security/faq.md
##########
@@ -0,0 +1,48 @@
+---
+title: Frequently Asked Questions
+weight: 5
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Frequently Asked Questions
+
+### During a security analysis of Flink, I noticed that Flink allows for remote code execution. Is this an issue?
Review comment:
I already said this offline but I don't think this should be in the docs.
It is independent from a given Flink version, and we purposefully put it on the main page to make it easier to discover.
The last thing we want is for a user/researcher to miss an FAQ item because they weren't looking at the latest and greatest documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
zentol commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r807901125
##########
File path: docs/content/docs/deployment/security/updates.md
##########
@@ -0,0 +1,73 @@
+---
Review comment:
That's decided on a case-by-case basis.
Generally speaking we aim to publish vulnerabilities once a mitigation is in place (either a new release or patch, depending on the severity).
In the case of Log4j we communicate it right away because the cat was out of the bag already at that point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r811832553
##########
File path: docs/content/docs/deployment/security/overview.md
##########
@@ -0,0 +1,68 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Security Overview
+
+Frameworks that process data are sensitive components; you must use authentication and encryption to
+secure your data and data sources. Apache Flink supports authentication with [Kerberos](https://web.mit.edu/kerberos/)
+and can be configured to encrypt all network communication with [SSL](https://www.ssl.com/faqs/faq-what-is-ssl/).
Review comment:
Apparently, based on org.apache.flink.configuration.SecurityOptions#SSL_PROTOCOL, we do use TLS by default but users can go back to SSL if they want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r811874959
##########
File path: docs/content/docs/deployment/security/ssl.md
##########
@@ -0,0 +1,243 @@
+---
+title: "Encryption and Authentication using SSL"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster](#).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+Most of these connections are exposed via REST/HTTP endpoints (and used by the web UI). Some external
+services used as sources or sinks may use some other network protocol.
+
+The server will, by default, accept connections from any client, meaning that the REST endpoint does
+not authenticate the client. These REST endpoints, however, can be configured to require SSL encryption
+and mutual authentication.
+
+However, the recommended approach is setting up and configuring a dedicated proxy service (a "sidecar
+proxy") that controls access to the REST endpoint. This involves binding the REST endpoint to the
+loopback interface (or the pod-local interface in Kubernetes) and starting a REST proxy that authenticates
+and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/)
+or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html).
+
+The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety
+of authentication options and thus better integration into existing infrastructures.
+
+## Queryable State
+
+Connections to the [queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}})
+endpoints is currently not authenticated or encrypted.
+
+## SSL Setups
+
+{{< img src="/fig/ssl_mutual_auth.svg" alt="SSL Mutual Authentication" width=75% >}}
+
+Each participant has a keystore and a truststore, which are files.
+
+A keystore contains a certificate (which contains a public key) and a private key. A truststore
+contains trusted certificates and certificate chains/authorities.
+
+Establishing encrypted, authenticated communication is a multi-step process, shown in the figure.
+Certificates are exchanged and validated against the truststore, after which the two parties can
+safely communicate.
+
+### Typical SSL Setup in Flink
+
+For mutually authenticated internal connections, note that:
+
+- a keystore and a truststore can contain the same dedicated certificate
+- the same file can be used for both keystore and truststore
+- wildcard hostnames or addresses can be used
+
+For internal communication between servers in a Flink cluster, a secure setup can be established with
+a single, self-signed certificate that all parties use as both their keystore and truststore. You can
+also use this approach for external communication when establishing mutual authentication for communication
+between clients and the Flink Master.
+
+### Configuring Keystores and Truststores
+
+The SSL configuration requires configuring a keystore and a truststore such that the truststore trusts
+the keystore's certificate.
+
+You can use the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html)
+to generate keys, certificates, keystores, and truststores:
+
+```bash
+ keytool -genkeypair -alias flink.internal -keystore internal.keystore \
+ -dname "CN=flink.internal" -storepass internal_store_password -keyalg RSA \
+ -keysize 4096 -storetype PKCS12
+```
+
+| Deployment mode | How to add the files |
+|------------------------|-------------------------------------------------------------------------|
+| Standalone clusters | copy the files to each node, or add them to a shared mounted filesystem |
+| Containerized clusters | add the files to the container images |
+| YARN | the cluster deployment phase can distribute these files |
+
+### Using Cipher Suites
+
+While the acts of encryption and decryption themselves are performed by keys, cipher suites outline
+the set of steps that the keys must follow to do so and the order in which these steps are executed.
+There are numerous cipher suites out there, each one with varying instructions on the encryption and
+decryption process.
+
+{{< hint warning >}}
+The [IETF RFC 7525](https://tools.ietf.org/html/rfc7525) recommends using a specific set of cipher
+suites for strong security. Since these cipher suites are not available on many setups out-of-the-box,
+Flink defaults to TLS_RSA_WITH_AES_128_CBC_SHA (a slightly weaker but more widely available cipher suite).
Review comment:
This is not related to this PR but I think it's bad from user perspective. The mentioned decision aims for compatibility and not for security. In other systems this is resolved in a way that nothing is provided by default which translates to the JVM provided default cipher suites. This way the user is going to use the strongest cipher suite which is available by default and not falling back to an old one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r811782136
##########
File path: docs/content/docs/deployment/security/ssl.md
##########
@@ -0,0 +1,243 @@
+---
+title: "Encryption and Authentication using SSL"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster](#).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+Most of these connections are exposed via REST/HTTP endpoints (and used by the web UI). Some external
+services used as sources or sinks may use some other network protocol.
+
+The server will, by default, accept connections from any client, meaning that the REST endpoint does
+not authenticate the client. These REST endpoints, however, can be configured to require SSL encryption
+and mutual authentication.
+
+However, the recommended approach is setting up and configuring a dedicated proxy service (a "sidecar
+proxy") that controls access to the REST endpoint. This involves binding the REST endpoint to the
+loopback interface (or the pod-local interface in Kubernetes) and starting a REST proxy that authenticates
+and forwards the requests to Flink. Examples for proxies that Flink users have deployed are [Envoy Proxy](https://www.envoyproxy.io/)
+or [NGINX with MOD_AUTH](http://nginx.org/en/docs/http/ngx_http_auth_request_module.html).
+
+The rationale behind delegating authentication to a proxy is that such proxies offer a wide variety
+of authentication options and thus better integration into existing infrastructures.
+
+## Queryable State
+
+Connections to the [queryable state]({{< ref "docs/dev/datastream/fault-tolerance/queryable_state" >}})
+endpoints is currently not authenticated or encrypted.
+
+## SSL Setups
+
+{{< img src="/fig/ssl_mutual_auth.svg" alt="SSL Mutual Authentication" width=75% >}}
+
+Each participant has a keystore and a truststore, which are files.
+
+A keystore contains a certificate (which contains a public key) and a private key. A truststore
+contains trusted certificates and certificate chains/authorities.
+
+Establishing encrypted, authenticated communication is a multi-step process, shown in the figure.
+Certificates are exchanged and validated against the truststore, after which the two parties can
+safely communicate.
+
+### Typical SSL Setup in Flink
+
+For mutually authenticated internal connections, note that:
+
+- a keystore and a truststore can contain the same dedicated certificate
+- the same file can be used for both keystore and truststore
+- wildcard hostnames or addresses can be used
+
+For internal communication between servers in a Flink cluster, a secure setup can be established with
+a single, self-signed certificate that all parties use as both their keystore and truststore. You can
+also use this approach for external communication when establishing mutual authentication for communication
+between clients and the Flink Master.
+
+### Configuring Keystores and Truststores
+
+The SSL configuration requires configuring a keystore and a truststore such that the truststore trusts
+the keystore's certificate.
+
+You can use the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html)
+to generate keys, certificates, keystores, and truststores:
+
+```bash
+ keytool -genkeypair -alias flink.internal -keystore internal.keystore \
+ -dname "CN=flink.internal" -storepass internal_store_password -keyalg RSA \
+ -keysize 4096 -storetype PKCS12
+```
+
+| Deployment mode | How to add the files |
+|------------------------|-------------------------------------------------------------------------|
+| Standalone clusters | copy the files to each node, or add them to a shared mounted filesystem |
+| Containerized clusters | add the files to the container images |
+| YARN | the cluster deployment phase can distribute these files |
+
+### Using Cipher Suites
+
+While the acts of encryption and decryption themselves are performed by keys, cipher suites outline
+the set of steps that the keys must follow to do so and the order in which these steps are executed.
+There are numerous cipher suites out there, each one with varying instructions on the encryption and
+decryption process.
+
+{{< hint warning >}}
+The [IETF RFC 7525](https://tools.ietf.org/html/rfc7525) recommends using a specific set of cipher
+suites for strong security. Since these cipher suites are not available on many setups out-of-the-box,
+Flink defaults to TLS_RSA_WITH_AES_128_CBC_SHA (a slightly weaker but more widely available cipher suite).
Review comment:
"If we can't come up w/ a better suite then maybe a warning can be printed that this is the default and weak..."
Isn't that what is already being said in that snippet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038807139
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit d46bd54e8d4e05e791d1679c5305100ec614c4c1 (Mon Feb 14 08:43:14 UTC 2022)
**Warnings:**
* Documentation files were touched, but no `docs/content.zh/` files: Update Chinese documentation or file Jira ticket.
* **Invalid pull request title: No valid Jira ID provided**
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91c09fb42686497a7e80405effd8fe474c24d4df Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r807903776
##########
File path: docs/content/docs/deployment/security/updates.md
##########
@@ -0,0 +1,73 @@
+---
Review comment:
Thanks for the explanation! I agree that Log4j was really an exception many ways.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4f76a841937d005508c859a643c6ac4c0fcbf7f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] infoverload commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
infoverload commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r809120538
##########
File path: docs/content/docs/deployment/security/running-cluster.md
##########
@@ -0,0 +1,292 @@
+---
+title: Incorporating Security Features in a Running Cluster
+weight: 4
+type: docs
+aliases:
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Incorporating Security Features in a Running Cluster
+
+This guide describes how Flink security works in the context of various [deployment modes]({{< ref "docs/deployment/overview" >}}),
+[file systems]({{< ref "docs/deployment/filesystems/overview" >}}), [connectors]({{< ref "docs/connectors" >}}),
+and [state backends]({{< ref "docs/ops/state/state_backends" >}}).
+
+## Running a Secure Flink Cluster in Standalone Mode
+
+Standalone mode is the most barebone way of deploying a Flink cluster where Flink processes are launched
+as processes on the operating system. You can learn more [here]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on all cluster nodes.
+2. Ensure that the Kerberos keytab file exists at the path indicated by the `security.kerberos.login.keytab`
+ setting on all cluster nodes.
+3. Deploy the Flink cluster as normal.
+
+You need to have an appropriate Java keystore and truststore accessible from each node in the Flink
+cluster. For standalone setups, this means copying the files to each node, or adding them to a shared
+mounted directory. For externally facing REST endpoints, the common name or subject alternative names
+in the certificate should match the node's hostname and IP address.
+
+Keystores and truststores can be generated using the [keytool utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html).
+
+## Running a Secure Flink Cluster in Kubernetes
+
+Flink’s native Kubernetes integration allows you to directly deploy Flink on a running Kubernetes cluster.
+You can learn more [here]({{< ref "docs/deployment/resource-providers/native_kubernetes" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on the client node.
+2. Ensure that the Kerberos keytab file exists at the path as indicated by the `security.kerberos.login.keytab`
+ on the client node.
+3. Deploy the Flink cluster as normal.
+
+### Setting up Kerberos
+
+When deploying Flink on Kubernetes, the Kerberos keytab file is automatically copied from the client
+node to the Flink containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file can be
+either fetched from the cluster environment or uploaded by Flink. In the latter case, you need to
+configure the `security.kerberos.krb5-conf.path` setting to indicate the path of the Kerberos configuration
+file and Flink will copy this file to its containers/pods.
+
+### Setting up keystores and truststores
+
+You need to have an appropriate Java keystore and truststore accessible from each node in the Flink
+cluster. For container-based setups, this means adding the keystore and truststore files to the container
+images. For externally facing REST endpoints, the common name or subject alternative names in the
+certificate should match the node's hostname and IP address.
+
+## Running a Secure Flink Cluster in YARN
+
+Flink services are submitted to YARN’s ResourceManager, which spawns containers on machines managed
+by YARN NodeManagers. Flink deploys its JobManager and TaskManager instances into such containers.
+You can learn more [here]({{< ref "docs/deployment/resource-providers/yarn" >}}).
+
+Incorporating security in this mode involves the following steps:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on the client node.
+2. Ensure that the Kerberos keytab file exists at the path as indicated by the `security.kerberos.login.keytab`
+ on the client node.
+3. Deploy the Flink cluster as normal.
+
+### Setting up Kerberos
+
+When deploying Flink on YARN, the Kerberos keytab is automatically copied from the client node to the
+Flink containers.
+
+To enable Kerberos authentication, the Kerberos configuration file is also required. This file can be
+either fetched from the cluster environment or uploaded by Flink. In the latter case, you need to
+configure the `security.kerberos.krb5-conf.path` setting to indicate the path of the Kerberos configuration
+file and Flink will copy this file to its containers/pods.
+
+#### Using `kinit` and without keytabs
+
+[`kinit`](https://web.mit.edu/kerberos/krb5-1.12/doc/user/user_commands/kinit.html) is used to obtain
+and cache Kerberos ticket-granting tickets.
+
+With YARN, it is possible to deploy a secure Flink cluster without a Kerberos keytab, which avoids the
+complexity of generating a keytab and entrusting the cluster manager with it. In this scenario, the
+ticket-granting-ticket cache, which is managed by `kinit`, is used and the Flink CLI acquires Hadoop
+delegation tokens (for HDFS and for HBase) instead. The main disadvantage is that the cluster is necessarily
+short-lived since the generated delegation tokens will expire (typically within a week).
+
+These are the steps to run a secure Flink cluster using `kinit`:
+1. Add [security-related configuration options]({{< ref "docs/deployment/config" >}}#auth-with-external-systems))
+ to the Flink configuration file (`flink-conf.yaml`) on the client node.
+2. Log in using the `kinit` command.
+3. Deploy the Flink cluster as normal.
+
+### Setting up keystores and truststores
+
+You need to have an appropriate Java keystore and truststore accessible from each node in the Flink
+cluster. For YARN setups, the cluster deployment phase can automatically distribute the keystore and
+truststore files. For externally facing REST endpoints, the common name or subject alternative names in the
+certificate should match the node's hostname and IP address.
+
+For the externally facing REST endpoint, the common name or subject alternative names in the certificate
+should match the node's hostname and IP address.
+
+For more information, see the [documentation on YARN security](https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnApplicationSecurity.md).
+
+## Setting Up SSL
+
+The easiest way to set up SSL is to generate a dedicated public/private key pair and self-signed
+certificate for the Flink deployment. The keystore and truststore are identical and contains
+only that key pair and certificate.
+
+In an environment where operators are constrained to use firm-wide Internal Certificate Authorities
+(cannot generate self-signed certificates), we recommend to still have a dedicated key pair and certificate
+for the Flink deployment, signed by that CA. However, the truststore must then also contain the
+CA's public certificate to accept the deployment's certificate during the SSL handshake (a requirement
+in the JDK truststore implementation).
+
+{{< hint danger >}}
+It is critical that you specify the fingerprint of the deployment certificate (via the `security.ssl.internal.cert.fingerprint`
+setting), when it is not self-signed. This is to pin that certificate as the only trusted certificate
+and prevent the truststore from trusting all certificates signed by that CA.
+{{< /hint >}}
+
+{{< hint info >}}
+Since internal connections are mutually authenticated with shared certificates, Flink can skip
+hostname verification. This makes container-based setups easier.
+{{< /hint >}}
+
+### Tips for a YARN Deployment
+
+To secure the external REST endpoint, you need to issue the REST endpoint's certificate such that it
+is valid for all hosts that the JobManager may get deployed to. This can be done with a wildcard DNS
+name or by adding multiple DNS names.
+
+The easiest way to deploy keystores and truststore is via the YARN client's *ship files* option (`-yt`).
+Copy the keystore and truststore files into a local directory (i.e. `deploy-keys/`) and start the YARN
+session like this:
+
+```bash
+flink run -m yarn-cluster -yt deploy-keys/ flinkapp.jar
+```
+
+When deployed using YARN, Flink's web UI is accessible through the YARN proxy's "Tracking URL". To
+ensure that the YARN proxy is able to access Flink's HTTPS URL, you need to configure the YARN proxy
+to accept Flink's SSL certificates. To do this, add the custom CA certificate into Java's default
+truststore on the YARN proxy node.
+
+### Example SSL Setup for Standalone Deployment and Kubernetes Deployment
Review comment:
I do not know. I got it from https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/security/security-ssl/#example-ssl-setup-standalone-and-kubernetes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] alpinegizmo commented on a change in pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
alpinegizmo commented on a change in pull request #18746:
URL: https://github.com/apache/flink/pull/18746#discussion_r809204892
##########
File path: docs/content/docs/deployment/security/ssl.md
##########
@@ -0,0 +1,243 @@
+---
+title: "Encryption and Authentication using SSL"
+weight: 3
+type: docs
+aliases:
+ - /deployment/security/ssl.html
+ - /ops/security-ssl.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Encryption and Authentication using SSL
+
+Flink supports mutual authentication (when two parties authenticate each other at the same time) and
+encryption of network communication with SSL for internal and external communication.
+
+**By default, SSL/TLS authentication and encryption is not enabled** (to have defaults work out-of-the-box).
+
+This guide will explain internal vs external connectivity, and provide instructions on how to enable
+SSL/TLS authentication and encryption for network communication with and between Flink processes. We
+will go through steps such as generating certificates, setting up TrustStores and KeyStores, and
+configuring cipher suites.
+
+For how-tos and tips for different deployment environments (i.e. standalone clusters, Kubernetes, YARN),
+check out the section on [Incorporating Security Features in a Running Cluster](#).
+
+## Internal and External Communication
+
+There are two types of network connections to authenticate and encrypt: internal and external.
+
+{{< img src="/fig/ssl_internal_external.svg" alt="Internal and External Connectivity" width=75% >}}
+
+For more flexibility, security for internal and external connectivity can be enabled and configured
+separately.
+
+### Internal Connectivity
+
+Flink internal communication refers to all connections made between Flink processes. These include:
+
+- Control messages: RPC between JobManager / TaskManager / Dispatcher / ResourceManager
+- Transfers on the data plane: connections between TaskManagers to exchange data during shuffles,
+ broadcasts, redistribution, etc
+- Blob service communication: distribution of libraries and other artifacts
+
+All internal connections are SSL authenticated and encrypted. The connections use **mutual authentication**,
+meaning both server and client side of each connection need to present the certificate to each other.
+The certificate acts as a shared secret and can be embedded into container images or attached to your
+deployment setup. These connections run Flink custom protocols. Users never connect directly to internal
+connectivity endpoints.
+
+### External Connectivity
+
+Flink external communication refers to all connections made from the outside to Flink processes.
+This includes:
+- communication with the Dispatcher to submit Flink jobs (session clusters)
+- communication of the Flink CLI with the JobManager to inspect and modify a running Flink job/application
+
+Most of these connections are exposed via REST/HTTP endpoints (and used by the web UI). Some external
Review comment:
AFAIK @knaufk is correct, and it would be more accurate to say "These connections are exposed via REST/HTTP endpoints (and are used by Flink's web UI). Some application-specific external services such as sources and sinks may use some other network protocol."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c4f76a841937d005508c859a643c6ac4c0fcbf7f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547)
* bc4c3c9c52616fe913872e5396bcef703c9ce45e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc4c3c9c52616fe913872e5396bcef703c9ce45e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720)
* 9f510145df68812e0eed11126fa839e85dedd26f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542",
"triggerID" : "bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6",
"triggerType" : "PUSH"
}, {
"hash" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545",
"triggerID" : "ec06532a2ea9edb00c82ce9304057966e5ef9ae7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec06532a2ea9edb00c82ce9304057966e5ef9ae7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #18746:
URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31386",
"triggerID" : "d46bd54e8d4e05e791d1679c5305100ec614c4c1",
"triggerType" : "PUSH"
}, {
"hash" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31444",
"triggerID" : "91c09fb42686497a7e80405effd8fe474c24d4df",
"triggerType" : "PUSH"
}, {
"hash" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31547",
"triggerID" : "c4f76a841937d005508c859a643c6ac4c0fcbf7f",
"triggerType" : "PUSH"
}, {
"hash" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31720",
"triggerID" : "bc4c3c9c52616fe913872e5396bcef703c9ce45e",
"triggerType" : "PUSH"
}, {
"hash" : "9f510145df68812e0eed11126fa839e85dedd26f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32055",
"triggerID" : "9f510145df68812e0eed11126fa839e85dedd26f",
"triggerType" : "PUSH"
}, {
"hash" : "8287d2291eb35027698751ab3a6519669ce59da9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540",
"triggerID" : "8287d2291eb35027698751ab3a6519669ce59da9",
"triggerType" : "PUSH"
}, {
"hash" : "112a4f6d047be628b5e0bc34ba54d246aea7f4c9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "112a4f6d047be628b5e0bc34ba54d246aea7f4c9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8287d2291eb35027698751ab3a6519669ce59da9 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32540)
* 112a4f6d047be628b5e0bc34ba54d246aea7f4c9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org