You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2020/11/03 16:59:08 UTC
[airflow] branch v1-10-test updated (470f242 -> df23b1f)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 470f242 Revert "Adds --no-rbac-ui flag for Breeze airflow 1.10 installation (#11315)"
new 44a93fd Revert "Update yamllint & isort pre-commit hooks (#11252)"
new 7c3f00d Enforce strict rules for yamllint (#11709)
new df23b1f Better file extension for Helm template (#11702)
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/build-images-workflow-run.yml | 24 +++---
.github/workflows/ci.yml | 2 +-
.github/workflows/codeql-analysis.yml | 2 +-
.github/workflows/codeql-cancel.yml | 4 +-
.github/workflows/scheduled_quarantined.yml | 2 +-
.pre-commit-config.yaml | 54 +++++++-------
chart/Chart.yaml | 1 +
chart/dockerfiles/statsd-exporter/mappings.yml | 2 +-
...yaml => pod-template-file.kubernetes-helm-yaml} | 0
chart/requirements.yaml | 2 +-
chart/templates/configmap.yaml | 2 +-
chart/values.yaml | 86 +++++++++++++---------
scripts/ci/kubernetes/ci_run_helm_testing.sh | 2 +-
tests/kubernetes/basic_pod.yaml | 2 +-
14 files changed, 101 insertions(+), 84 deletions(-)
rename chart/files/{pod-template-file.yaml => pod-template-file.kubernetes-helm-yaml} (100%)
[airflow] 01/03: Revert "Update yamllint & isort pre-commit hooks
(#11252)"
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 44a93fd2be4d55fd21ceab7d7a36a6f9c95e5b98
Author: Kaxil Naik <ka...@gmail.com>
AuthorDate: Tue Nov 3 16:41:13 2020 +0000
Revert "Update yamllint & isort pre-commit hooks (#11252)"
This reverts commit 6426e92c3f64821c6d2426f597aa4f78009a57c0.
---
.pre-commit-config.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index eee592e..fd243ec 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -157,7 +157,7 @@ repos:
- id: rst-backticks
- id: python-no-log-warn
- repo: https://github.com/adrienverge/yamllint
- rev: v1.25.0
+ rev: v1.24.2
hooks:
- id: yamllint
name: Check yaml files with yamllint
[airflow] 03/03: Better file extension for Helm template (#11702)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit df23b1fc4e1c274a72c8522fcdfd75fa740e2e3e
Author: Kamil Breguła <mi...@users.noreply.github.com>
AuthorDate: Wed Oct 21 12:17:19 2020 +0200
Better file extension for Helm template (#11702)
* Better file extension for Helm template
* fixup! Better file extension for Helm template
(cherry picked from commit ffc9aebeb265b7b2f71855e539b20895df2f4ca3)
---
.../{pod-template-file.yaml => pod-template-file.kubernetes-helm-yaml} | 0
chart/templates/configmap.yaml | 2 +-
scripts/ci/kubernetes/ci_run_helm_testing.sh | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/chart/files/pod-template-file.yaml b/chart/files/pod-template-file.kubernetes-helm-yaml
similarity index 100%
rename from chart/files/pod-template-file.yaml
rename to chart/files/pod-template-file.kubernetes-helm-yaml
diff --git a/chart/templates/configmap.yaml b/chart/templates/configmap.yaml
index cc9a388..d5b4b08 100644
--- a/chart/templates/configmap.yaml
+++ b/chart/templates/configmap.yaml
@@ -60,6 +60,6 @@ data:
{{- if .Values.podTemplate }}
{{ .Values.podTemplate | nindent 4 }}
{{- else }}
-{{ tpl (.Files.Get "files/pod-template-file.yaml") . | nindent 4 }}
+{{ tpl (.Files.Get "files/pod-template-file.kubernetes-helm-yaml") . | nindent 4 }}
{{- end }}
{{- end }}
diff --git a/scripts/ci/kubernetes/ci_run_helm_testing.sh b/scripts/ci/kubernetes/ci_run_helm_testing.sh
index 224cc9e..93d8337 100755
--- a/scripts/ci/kubernetes/ci_run_helm_testing.sh
+++ b/scripts/ci/kubernetes/ci_run_helm_testing.sh
@@ -20,7 +20,7 @@ echo "Running helm tests"
chart_directory="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/../../../chart/"
-cat chart/files/pod-template-file.yaml > chart/templates/pod-template-file.yaml
+cat chart/files/pod-template-file.kubernetes-helm-yaml > chart/templates/pod-template-file.yaml
docker run -w /airflow-chart -v "$chart_directory":/airflow-chart \
--entrypoint /bin/sh \
[airflow] 02/03: Enforce strict rules for yamllint (#11709)
Posted by ka...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7c3f00d2de98233279c3b0892eafa1d6ae90b684
Author: Kamil Breguła <mi...@users.noreply.github.com>
AuthorDate: Wed Oct 21 12:24:32 2020 +0200
Enforce strict rules for yamllint (#11709)
---
.github/workflows/build-images-workflow-run.yml | 24 +++----
.github/workflows/ci.yml | 2 +-
.github/workflows/codeql-analysis.yml | 2 +-
.github/workflows/codeql-cancel.yml | 4 +-
.github/workflows/scheduled_quarantined.yml | 2 +-
.pre-commit-config.yaml | 52 +++++++--------
chart/Chart.yaml | 1 +
chart/dockerfiles/statsd-exporter/mappings.yml | 2 +-
chart/requirements.yaml | 2 +-
chart/values.yaml | 86 +++++++++++++++----------
tests/kubernetes/basic_pod.yaml | 2 +-
11 files changed, 98 insertions(+), 81 deletions(-)
diff --git a/.github/workflows/build-images-workflow-run.yml b/.github/workflows/build-images-workflow-run.yml
index b791fc3..2ad2931 100644
--- a/.github/workflows/build-images-workflow-run.yml
+++ b/.github/workflows/build-images-workflow-run.yml
@@ -17,7 +17,7 @@
#
---
name: "Build Images"
-on:
+on: # yamllint disable-line rule:truthy
workflow_run:
workflows: ["CI Build"]
types: ['requested']
@@ -59,13 +59,13 @@ jobs:
if: github.repository == 'apache/airflow' || github.event.workflow_run.event != 'schedule'
steps:
- name: "Get information about the origin 'CI Build' run"
- uses: potiuk/get-workflow-origin@c657bb36aef4a7402bbe9b2e09a820320f8ff447 # v1
+ uses: potiuk/get-workflow-origin@c657bb36aef4a7402bbe9b2e09a820320f8ff447 # v1
id: source-run-info
with:
token: ${{ secrets.GITHUB_TOKEN }}
sourceRunId: ${{ github.event.workflow_run.id }}
- name: "Cancel duplicated 'CI Build' runs"
- uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: duplicates
@@ -84,7 +84,7 @@ jobs:
# in GitHub Actions, we have to use Job names to match Event/Repo/Branch from the
# build-info step there to find the duplicates ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -105,7 +105,7 @@ jobs:
# can cancel all the matching "Build Images" workflow runs in the two following steps.
# Yeah. Adding to the complexity ¯\_(ツ)_/¯.
- uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
id: cancel-failed
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -140,7 +140,7 @@ jobs:
# it to cancel any jobs that have matching names containing Source Run Id:
# followed by one of the run ids. Yes I know it's super complex ¯\_(ツ)_/¯.
if: env.BUILD_IMAGES == 'true' && steps.source-run-info-failed.outputs.cancelledRuns != '[]'
- uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
with:
cancelMode: namedJobs
token: ${{ secrets.GITHUB_TOKEN }}
@@ -225,8 +225,8 @@ jobs:
needs: [build-info, cancel-workflow-runs]
strategy:
matrix:
- # We need to attempt to build all possible versions here because workflow_run
- # event is run from master for both master and v1-10-tests
+ # We need to attempt to build all possible versions here because workflow_run
+ # event is run from master for both master and v1-10-tests
python-version: ${{ fromJson(needs.build-info.outputs.allPythonVersions) }}
image-type: [CI, PROD]
fail-fast: true
@@ -277,7 +277,7 @@ jobs:
echo "::set-output name=proceed::false"
fi
- name: Initiate Github Checks for Building image
- uses: LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 #v1.1.0
+ uses: LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 # v1.1.0
id: build-image-check
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -325,7 +325,7 @@ jobs:
run: ./scripts/ci/images/ci_push_production_images.sh
if: matrix.image-type == 'PROD' && steps.defaults.outputs.proceed == 'true'
- name: Update Github Checks for Building image with status
- uses: LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 #v1.1.0
+ uses: LouisBrunner/checks-action@9f02872da71b6f558c6a6f190f925dde5e4d8798 # v1.1.0
if: always() && steps.defaults.outputs.proceed == 'true'
with:
token: ${{ secrets.GITHUB_TOKEN }}
@@ -347,7 +347,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@cancel_message #v3
+ uses: potiuk/cancel-workflow-runs@cancel_message # v3
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
@@ -362,7 +362,7 @@ jobs:
needs: [build-images]
steps:
- name: "Canceling the 'CI Build' source workflow in case of failure!"
- uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
with:
token: ${{ secrets.GITHUB_TOKEN }}
cancelMode: self
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 85f8e27..46cdea3 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -17,7 +17,7 @@
#
---
name: CI Build
-on:
+on: # yamllint disable-line rule:truthy
schedule:
- cron: '28 0 * * *'
push:
diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml
index 01bac7e..40e518d 100644
--- a/.github/workflows/codeql-analysis.yml
+++ b/.github/workflows/codeql-analysis.yml
@@ -1,7 +1,7 @@
---
name: "CodeQL"
-on:
+on: # yamllint disable-line rule:truthy
push:
branches: [master]
pull_request:
diff --git a/.github/workflows/codeql-cancel.yml b/.github/workflows/codeql-cancel.yml
index 2221d7a..bba6cc7 100644
--- a/.github/workflows/codeql-cancel.yml
+++ b/.github/workflows/codeql-cancel.yml
@@ -1,6 +1,6 @@
---
name: "Cancel duplicate CodeQL workflows"
-on:
+on: # yamllint disable-line rule:truthy
workflow_run:
workflows: ["CodeQL"]
types: ['requested']
@@ -12,7 +12,7 @@ jobs:
if: github.repository == 'apache/airflow' || github.event.workflow_run.event != 'schedule'
steps:
- name: "Cancel duplicated 'CodeQL' runs"
- uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
+ uses: potiuk/cancel-workflow-runs@f696c622a83e4a63fff74848d3b149074658607d # v3_2
id: cancel
with:
token: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/scheduled_quarantined.yml b/.github/workflows/scheduled_quarantined.yml
index 5468943..b75627c 100644
--- a/.github/workflows/scheduled_quarantined.yml
+++ b/.github/workflows/scheduled_quarantined.yml
@@ -17,7 +17,7 @@
#
---
name: Quarantined Build
-on:
+on: # yamllint disable-line rule:truthy
schedule:
# Run quarantined builds 4 times a day to gather better quarantine stats
- cron: '12 */6 * * *'
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index fd243ec..f1b930a 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -161,34 +161,34 @@ repos:
hooks:
- id: yamllint
name: Check yaml files with yamllint
- entry: yamllint -c yamllint-config.yml
+ entry: yamllint -c yamllint-config.yml --strict
types: [yaml]
exclude: ^.*init_git_sync\.template\.yaml$|^.*airflow\.template\.yaml$|^chart/templates/.*\.yaml$
- ##
- ## Dear committer.
- ##
- ## If you ever come here to add the missing isort step here - hear a little warning.
- ##
- ## Initially isort will cause surprising duplicates of urlparse and other urllib related methods.
- ## The urllib imports seem broken for python 2 but they are actually fine due to future
- ## backport aliases installed elsewhere in the code (implicitly) - in 6 places.
- ##
- ## When you decide how to fix it (likely talking to other people in community) and you push
- ## build to CI you will find terrible truth that in Airflow 1.10 modules are so much
- ## cross-dependent, that imports in a number of places have to be done in specific order and
- ## if this is not followed properly, circular imports kick-in and you are doomed.
- ##
- ## Running isort breaks the import House of Cards and there is no easy way to fix it short of
- ## splitting a number of files and probably breaking compatibility.
- ##
- ## Luckily this has been fixed in Airflow 2.0 by proper untangling of the cross-dependencies and
- ## 1.10.* branch is really in maintenance mode, so do not really waste your time here.
- ##
- ## Unless you really want of course. But then either delete this comment or increase the counter
- ## below after you give up.
- ##
- ## Total hours wasted here = 3
- ##
+ ##
+ ## Dear committer.
+ ##
+ ## If you ever come here to add the missing isort step here - hear a little warning.
+ ##
+ ## Initially isort will cause surprising duplicates of urlparse and other urllib related methods.
+ ## The urllib imports seem broken for python 2 but they are actually fine due to future
+ ## backport aliases installed elsewhere in the code (implicitly) - in 6 places.
+ ##
+ ## When you decide how to fix it (likely talking to other people in community) and you push
+ ## build to CI you will find terrible truth that in Airflow 1.10 modules are so much
+ ## cross-dependent, that imports in a number of places have to be done in specific order and
+ ## if this is not followed properly, circular imports kick-in and you are doomed.
+ ##
+ ## Running isort breaks the import House of Cards and there is no easy way to fix it short of
+ ## splitting a number of files and probably breaking compatibility.
+ ##
+ ## Luckily this has been fixed in Airflow 2.0 by proper untangling of the cross-dependencies and
+ ## 1.10.* branch is really in maintenance mode, so do not really waste your time here.
+ ##
+ ## Unless you really want of course. But then either delete this comment or increase the counter
+ ## below after you give up.
+ ##
+ ## Total hours wasted here = 3
+ ##
- repo: local
hooks:
- id: shellcheck
diff --git a/chart/Chart.yaml b/chart/Chart.yaml
index 927b412..93de06e 100644
--- a/chart/Chart.yaml
+++ b/chart/Chart.yaml
@@ -16,6 +16,7 @@
# under the License.
# apiVersion v1 is Helm 2
+---
apiVersion: v1
name: airflow
version: 1.0.0
diff --git a/chart/dockerfiles/statsd-exporter/mappings.yml b/chart/dockerfiles/statsd-exporter/mappings.yml
index f86854e..f10ab00 100644
--- a/chart/dockerfiles/statsd-exporter/mappings.yml
+++ b/chart/dockerfiles/statsd-exporter/mappings.yml
@@ -14,7 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+---
mappings:
# Map dot separated stats to labels
- match: airflow.dagrun.dependency-check.*.*
diff --git a/chart/requirements.yaml b/chart/requirements.yaml
index 56f29ee..c0b6af2 100644
--- a/chart/requirements.yaml
+++ b/chart/requirements.yaml
@@ -14,7 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+---
dependencies:
- name: postgresql
version: 6.3.12
diff --git a/chart/values.yaml b/chart/values.yaml
index ba7a01e..b8c4fcf 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -223,12 +223,12 @@ webserver:
extraNetworkPolicies: []
resources: {}
- # limits:
- # cpu: 100m
- # memory: 128Mi
- # requests:
- # cpu: 100m
- # memory: 128Mi
+ # limits:
+ # cpu: 100m
+ # memory: 128Mi
+ # requests:
+ # cpu: 100m
+ # memory: 128Mi
# Create initial user.
defaultUser:
@@ -265,12 +265,12 @@ flower:
# Additional network policies as needed
extraNetworkPolicies: []
resources: {}
- # limits:
- # cpu: 100m
- # memory: 128Mi
- # requests:
- # cpu: 100m
- # memory: 128Mi
+ # limits:
+ # cpu: 100m
+ # memory: 128Mi
+ # requests:
+ # cpu: 100m
+ # memory: 128Mi
service:
type: ClusterIP
@@ -281,12 +281,12 @@ statsd:
# Additional network policies as needed
extraNetworkPolicies: []
resources: {}
- # limits:
- # cpu: 100m
- # memory: 128Mi
- # requests:
- # cpu: 100m
- # memory: 128Mi
+ # limits:
+ # cpu: 100m
+ # memory: 128Mi
+ # requests:
+ # cpu: 100m
+ # memory: 128Mi
service:
extraAnnotations: {}
@@ -313,13 +313,21 @@ pgbouncer:
config:
maxUnavailable: 1
+ # Limit the resources to pgbouncerExported.
+ # When you specify the resource request the scheduler uses this information to decide which node to place
+ # the Pod on. When you specify a resource limit for a Container, the kubelet enforces those limits so
+ # that the running container is not allowed to use more of that resource than the limit you set.
+ # See: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
+ # Example:
+ #
+ # resource:
+ # limits:
+ # cpu: 100m
+ # memory: 128Mi
+ # requests:
+ # cpu: 100m
+ # memory: 128Mi
resources: {}
- # limits:
- # cpu: 100m
- # memory: 128Mi
- # requests:
- # cpu: 100m
- # memory: 128Mi
service:
extraAnnotations: {}
@@ -365,11 +373,13 @@ redis:
registry:
secretName: ~
+ # Example:
+ # connection:
+ # user: ~
+ # pass: ~
+ # host: ~
+ # email: ~
connection: {}
- # user: ~
- # pass: ~
- # host: ~
- # email: ~
# Elasticsearch logging configuration
elasticsearch:
@@ -378,11 +388,14 @@ elasticsearch:
# A secret containing the connection
secretName: ~
# Or an object representing the connection
+ # Example:
+ # connection:
+ # user: ~
+ # pass: ~
+ # host: ~
+ # port: ~
connection: {}
- # user: ~
- # pass: ~
- # host: ~
- # port: ~
+
# All ports used by chart
ports:
@@ -559,7 +572,8 @@ dags:
# GIT_SYNC_USERNAME: <base64_encoded_git_username>
# GIT_SYNC_PASSWORD: <base64_encoded_git_password>
# and specify the name of the secret below
- #credentialsSecret: git-credentials
+ #
+ # credentialsSecret: git-credentials
#
#
# If you are using an ssh clone url, you can load
@@ -573,10 +587,12 @@ dags:
# # key needs to be gitSshKey
# gitSshKey: <base64_encoded_data>
# and specify the name of the secret below
- #sshKeySecret: airflow-ssh-secret
+ # sshKeySecret: airflow-ssh-secret
+ #
# If you are using an ssh private key, you can additionally
# specify the content of your known_hosts file, example:
- #knownHosts: |
+ #
+ # knownHosts: |
# <host1>,<ip1> <key1>
# <host2>,<ip2> <key2>
# interval between git sync attempts in seconds
diff --git a/tests/kubernetes/basic_pod.yaml b/tests/kubernetes/basic_pod.yaml
index 417b56d..fff84d1 100644
--- a/tests/kubernetes/basic_pod.yaml
+++ b/tests/kubernetes/basic_pod.yaml
@@ -14,7 +14,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
+---
apiVersion: v1
kind: Pod
metadata: