You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/20 16:12:42 UTC
[GitHub] [airflow] ashb opened a new pull request #11694: Create pytest fixtures for testing Helm chart templates
ashb opened a new pull request #11694:
URL: https://github.com/apache/airflow/pull/11694
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Example test case:
Compare: https://github.com/apache/airflow/blob/782b8b374f191b13fa971db6bae501661139ed10/chart/tests/git-sync-scheduler_test.yaml#L30-L51
```yaml
set:
dags:
gitSync:
enabled: true
containerName: git-sync-test
containerTag: test-tag
containerRepository: test-registry/test-repo
wait: 66
maxFailures: 70
subPath: "path1/path2"
dest: "test-dest"
root: "/git-root"
rev: HEAD
depth: 1
repo: https://github.com/apache/airflow.git
branch: test-branch
sshKeySecret: ~
credentialsSecret: ~
knownHosts: ~
persistence:
enabled: true
asserts:
- equal:
path: spec.template.spec.containers[1]
value:
name: git-sync-test
securityContext:
runAsUser: 65533
image: test-registry/test-repo:test-tag
env:
- name: GIT_SYNC_REV
value: HEAD
- name: GIT_SYNC_BRANCH
value: test-branch
- name: GIT_SYNC_REPO
value: https://github.com/apache/airflow.git
- name: GIT_SYNC_DEPTH
value: "1"
- name: GIT_SYNC_ROOT
value: /git-root
- name: GIT_SYNC_DEST
value: test-dest
- name: GIT_SYNC_ADD_USER
value: "true"
- name: GIT_SYNC_WAIT
value: "66"
- name: GIT_SYNC_MAX_SYNC_FAILURES
value: "70"
volumeMounts:
- mountPath: /git-root
name: dags
```
vs
```python
class TestGitSyncSchedulerDeployment:
"""
Basic tests of chart functionality
"""
DEFAULT_TEMPLATES = ['scheduler/scheduler-deployment.yaml']
def test_validate_git_sync_container_spec(self, render_chart):
resources = render_chart(values={
'dags': {
'gitSync': {
'enabled': True,
'containerName': 'git-sync-test',
'containerTag': 'test-tag',
'containerRepository': 'test-registry/test-repo',
'wait': 66,
'maxFailures': 70,
'subPath': 'path1/path2',
'dest': 'test-dest',
'root': '/git-root',
'rev': 'HEAD',
'depth': 1,
'repo': 'https://github.com/apache/airflow.git',
'branch': 'test-branch',
'sshKeySecret': None,
'credentialsSecret': None,
'knownHosts': None,
},
'persistence': {'enabled': True},
},
})
assert len(resources) == 1
resource = resources[0]
assert resource['spec']['template']['spec']['containers'][1] == {
'env': [
{'name': 'GIT_SYNC_REV', 'value': 'HEAD'},
{'name': 'GIT_SYNC_BRANCH', 'value': 'test-branch'},
{'name': 'GIT_SYNC_REPO', 'value': 'https://github.com/apache/airflow.git'},
{'name': 'GIT_SYNC_DEPTH', 'value': '1'},
{'name': 'GIT_SYNC_ROOT', 'value': '/git-root'},
{'name': 'GIT_SYNC_DEST', 'value': 'test-dest'},
{'name': 'GIT_SYNC_ADD_USER', 'value': 'true'},
{'name': 'GIT_SYNC_WAIT', 'value': '66'},
{'name': 'GIT_SYNC_MAX_SYNC_FAILURES', 'value': '70'},
],
'image': 'test-registry/test-repo:test-tag',
'name': 'git-sync-test',
'securityContext': {'runAsUser': 65533},
'volumeMounts': [{'mountPath': '/git-root', 'name': 'dags'}],
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed pull request #11694: Create pytest fixtures for testing Helm chart templates
Posted by GitBox <gi...@apache.org>.
ashb closed pull request #11694:
URL: https://github.com/apache/airflow/pull/11694
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11694: Create pytest fixtures for testing Helm chart templates
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11694:
URL: https://github.com/apache/airflow/pull/11694#discussion_r508942109
##########
File path: chart/tests/conftest.py
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import logging
+import os
+import subprocess
+from typing import Dict, List, Optional
+
+import pytest
+import yaml
+
+log = logging.getLogger(__name__)
+
+CHART_FOLDER = os.path.normpath(os.path.join(__file__, os.pardir, os.pardir))
+
+
+def run_cmd(cmd, **kwargs):
+ """
+ Helper method to run a command, and include any output in case of error.
+ """
+ try:
+ log.debug('Running %s', cmd)
+ return subprocess.check_output(
+ cmd,
+ stderr=subprocess.STDOUT,
+ **kwargs
+ )
+ except subprocess.CalledProcessError as e:
+ raise AssertionError(f'Command {cmd} failed with: {e.output.decode("utf-8")}')
+
+
+@pytest.fixture(scope="session", autouse=True)
+def initalize_chart():
+ """
+ Get the helm chart ready to install/render.
+
+ This is run for us automatically by pytest once per test-session
+ """
+ run_cmd(['helm', 'repo', 'add', 'stable', 'https://kubernetes-charts.storage.googleapis.com/'])
+ run_cmd(['helm', 'dep', 'update', CHART_FOLDER])
+
+
+@pytest.fixture
+def render_chart(request):
Review comment:
Yeah, in not sure either. I was basically looking at what we have in the yaml files and modelling this off of that.
I was thinking it's kind of nice to not have to specify the same file in every test function of a class as they will likely be testing the same files bit with different value inputs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11694: Create pytest fixtures for testing Helm chart templates
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11694:
URL: https://github.com/apache/airflow/pull/11694#discussion_r508859361
##########
File path: chart/tests/conftest.py
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import logging
+import os
+import subprocess
+from typing import Dict, List, Optional
+
+import pytest
+import yaml
+
+log = logging.getLogger(__name__)
+
+CHART_FOLDER = os.path.normpath(os.path.join(__file__, os.pardir, os.pardir))
+
+
+def run_cmd(cmd, **kwargs):
+ """
+ Helper method to run a command, and include any output in case of error.
+ """
+ try:
+ log.debug('Running %s', cmd)
+ return subprocess.check_output(
+ cmd,
+ stderr=subprocess.STDOUT,
+ **kwargs
+ )
+ except subprocess.CalledProcessError as e:
+ raise AssertionError(f'Command {cmd} failed with: {e.output.decode("utf-8")}')
+
+
+@pytest.fixture(scope="session", autouse=True)
+def initalize_chart():
+ """
+ Get the helm chart ready to install/render.
+
+ This is run for us automatically by pytest once per test-session
+ """
+ run_cmd(['helm', 'repo', 'add', 'stable', 'https://kubernetes-charts.storage.googleapis.com/'])
+ run_cmd(['helm', 'dep', 'update', CHART_FOLDER])
+
+
+@pytest.fixture
+def render_chart(request):
Review comment:
I don't know if I wouldn't prefer the templates used to be always explicit passed. Many of the contributions we have to the Helm Chart are small and made by many different contributors. I don't want them to have to learn new concepts if it's not needed. Tests for templates are already a high threshold for entering the project. This is not common in all projects.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11694: Create pytest fixtures for testing Helm chart templates
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11694:
URL: https://github.com/apache/airflow/pull/11694#discussion_r508859361
##########
File path: chart/tests/conftest.py
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import logging
+import os
+import subprocess
+from typing import Dict, List, Optional
+
+import pytest
+import yaml
+
+log = logging.getLogger(__name__)
+
+CHART_FOLDER = os.path.normpath(os.path.join(__file__, os.pardir, os.pardir))
+
+
+def run_cmd(cmd, **kwargs):
+ """
+ Helper method to run a command, and include any output in case of error.
+ """
+ try:
+ log.debug('Running %s', cmd)
+ return subprocess.check_output(
+ cmd,
+ stderr=subprocess.STDOUT,
+ **kwargs
+ )
+ except subprocess.CalledProcessError as e:
+ raise AssertionError(f'Command {cmd} failed with: {e.output.decode("utf-8")}')
+
+
+@pytest.fixture(scope="session", autouse=True)
+def initalize_chart():
+ """
+ Get the helm chart ready to install/render.
+
+ This is run for us automatically by pytest once per test-session
+ """
+ run_cmd(['helm', 'repo', 'add', 'stable', 'https://kubernetes-charts.storage.googleapis.com/'])
+ run_cmd(['helm', 'dep', 'update', CHART_FOLDER])
+
+
+@pytest.fixture
+def render_chart(request):
Review comment:
I don't know if I wouldn't prefer the templates used to be always explicit passed. Many of the contributions we have to the Helm Chart are small and made by many different contributors. I don't want them to have to learn new concepts if it's not needed. Tests for templates are already a high threshold for entering the project. This is not common in all projects.
This is not a strong opinion, and it can stay if you think otherwise.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org