You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/14 23:10:39 UTC

[GitHub] [airflow] gmsantos opened a new pull request, #23711: Helmchart: Provision Standalone Dag Processor

gmsantos opened a new pull request, #23711:
URL: https://github.com/apache/airflow/pull/23711

   Initial support to deploy standalone dag processor from Helm on Airflow 2.3+
   
   It is disabled by default for now (let me know if is preferred to enable it by default)
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jedcunningham commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r911331264


##########
chart/templates/scheduler/scheduler-deployment.yaml:
##########
@@ -141,7 +141,7 @@ spec:
           {{- include "custom_airflow_environment" . | indent 10 }}
           {{- include "standard_airflow_environment" . | indent 10 }}
         {{- end }}
-        {{- if .Values.dags.gitSync.enabled }}
+        {{- if and (not .Values.dagProcessor.enabled) .Values.dags.gitSync.enabled }}

Review Comment:
   Don't we still need this if we have dagProcessor and local?



##########
chart/templates/scheduler/scheduler-deployment.yaml:
##########
@@ -202,13 +202,13 @@ spec:
               subPath: airflow_local_settings.py
               readOnly: true
             {{- end }}
-            {{- if or .Values.dags.persistence.enabled .Values.dags.gitSync.enabled }}
-            {{- include "airflow_dags_mount" . | nindent 12 }}
-            {{- end }}
+{{- if and (or (not .Values.dagProcessor.enabled) $local) (or .Values.dags.persistence.enabled .Values.dags.gitSync.enabled) }}
+{{- include "airflow_dags_mount" . | nindent 12 }}
+{{- end }}
 {{- if .Values.scheduler.extraVolumeMounts }}
 {{ toYaml .Values.scheduler.extraVolumeMounts | indent 12 }}
 {{- end }}
-{{- if .Values.dags.gitSync.enabled }}
+{{- if and (not .Values.dagProcessor.enabled) .Values.dags.gitSync.enabled }}

Review Comment:
   Same here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jedcunningham commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r874206665


##########
chart/templates/dagprocessor/dag-processor-deployment.yaml:
##########
@@ -0,0 +1,214 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+################################
+## Airflow Dag Processor Deployment
+#################################
+{{- if semverCompare ">=2.3.0" .Values.airflowVersion }}
+{{- if .Values.dagProcessor.enabled }}
+{{- $nodeSelector := or .Values.dagProcessor.nodeSelector .Values.nodeSelector }}
+{{- $affinity := or .Values.dagProcessor.affinity .Values.affinity }}
+{{- $tolerations := or .Values.dagProcessor.tolerations .Values.tolerations }}
+{{- $topologySpreadConstraints := or .Values.dagProcessor.topologySpreadConstraints .Values.topologySpreadConstraints }}
+{{- $securityContext := include "airflowSecurityContext" (list . .Values.dagProcessor) }}
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: {{ .Release.Name }}-dag-processor
+  labels:
+    tier: airflow
+    component: dag-processor

Review Comment:
   That second one sounds good to me.



##########
tests/charts/test_basic_helm_chart.py:
##########
@@ -97,6 +97,78 @@ def test_basic_deployments(self):
                 "TEST-LABEL"
             ), f"Missing label TEST-LABEL on {k8s_name}. Current labels: {labels}"
 
+    def test_basic_deployment_with_standalone_scheduler(self):
+        k8s_objects = render_chart(
+            "TEST-BASIC",
+            values={
+                "chart": {
+                    'metadata': 'AA',
+                },
+                'labels': {"TEST-LABEL": "TEST-VALUE"},
+                "fullnameOverride": "TEST-BASIC",
+                "dagProcessor": {'enabled': True},

Review Comment:
   I think being disabled by default at this point makes sense.



##########
chart/templates/_helpers.yaml:
##########
@@ -653,6 +664,14 @@ Create the name of the cleanup service account to use
     airflow jobs check --job-type TriggererJob --hostname $(hostname)
 {{- end }}
 
+{{define "dag_processor_liveness_check_command"}}
+  - sh
+  - -c
+  - |
+    CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
+    airflow jobs check --job-type SchedulerJob --hostname $(hostname)

Review Comment:
   In the meantime, you can probably leave off the job type and it'll still work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1156744620

   NOOOOOOO ! We found a root cause it was not you @gmsantos !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1156296018

   I re-run failing jobs, but if they fail again - they do not look like false positives - likely a bug in the helm modifications


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1157529433

   Something still broke with LocalExecutor, it is returning 404 for some example DAGs. Maybe it's related with some changes form this PR.
   
   I will check that later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1157554848

   > Something still broke with LocalExecutor, it is returning 404 for some example DAGs. Maybe it's related with some changes form this PR.
   
   I think this time yeah, it's not false negative. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] github-actions[bot] commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1133883753

   The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1156606574

   I'll to reproduce it locally


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jtvmatos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
jtvmatos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1251036264

   Hi @gmsantos,
   
   Sorry the mention in a merged PR, but taking into account the new `Support multiple DagProcessors parsing files from different locations` (https://github.com/apache/airflow/pull/25935), will it be possible to use this new Helm configurations to setup two DagProcessors for different subdirs?  Or should I create a new issue/PR?
   
   Thanks!
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r873084653


##########
tests/charts/test_dag_processor.py:
##########
@@ -0,0 +1,431 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+import jmespath
+from parameterized import parameterized
+
+from tests.charts.helm_template_generator import render_chart
+
+
+class DagProcessorTest(unittest.TestCase):
+    @parameterized.expand(
+        [
+            ("2.2.0", 0),
+            ("2.3.0", 1),
+        ]
+    )
+    def test_only_exists_on_new_airflow_versions(self, airflow_version, num_docs):
+        """Standalone Dag Processor was only added from Airflow 2.3 onwards"""
+        docs = render_chart(
+            values={
+                "airflowVersion": airflow_version,
+                "dagProcessor": {"enabled": True},
+            },
+            show_only=["templates/dagprocessor/dag-processor-deployment.yaml"],
+        )
+
+        assert num_docs == len(docs)
+
+    def test_can_be_disabled(self):
+        """
+        Standalone Dag Processor is disabled by default.
+        """
+        docs = render_chart(
+            values={"dagProcessor": {"enabled": False}},
+            show_only=["templates/dagprocessor/dag-processor-deployment.yaml"],
+        )
+
+        assert 0 == len(docs)
+
+    def test_disable_wait_for_migration(self):
+        docs = render_chart(
+            values={
+                "dagProcessor": {
+                    "enabled": True,

Review Comment:
   Same here, if we enable dagProcessor by default, we can remove it from the test values



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1148590400

   hey @jedcunningham, it should be ready to review. I just trying to make all the PR checks pass ✅ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1150324330

   @jedcunningham I could use some help to retry the failed CI checks here, please :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r914215373


##########
chart/templates/scheduler/scheduler-deployment.yaml:
##########
@@ -141,7 +141,7 @@ spec:
           {{- include "custom_airflow_environment" . | indent 10 }}
           {{- include "standard_airflow_environment" . | indent 10 }}
         {{- end }}
-        {{- if .Values.dags.gitSync.enabled }}
+        {{- if and (not .Values.dagProcessor.enabled) .Values.dags.gitSync.enabled }}

Review Comment:
   Makes sense, local executor still requires DAGs to be mounted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1165007592

   @jedcunningham I got some progress here. The change that is failing the tests happens in this [change](https://github.com/apache/airflow/pull/23711/commits/2eaca8f2e390dd6e3c8874b4847d416f83c07d9b ) to include the dag_mount.
   
   I think in some cases we do not need the dag mount in the scheduler:
   
   - [1] if the DAGs are loaded from examples or baked with the Airflow image (no matter what, the DAGs will be available to the scheduler)
   - [2] when the DAG processor is enabled and we are not using a local executor, neither use DAGs persistence or gitSync (the mount is moved to dag processor deployment then)
   - [2] when the DAG processor is enabled and we are using a local executor, but we neither use DAGs persistence or gitSync (it falls on the case [1], DAGs are already here)
   
   We need to mount the DAGs to the scheduler when:
   - [3] DAG processor is disabled, and we use either DAGs persistence or gitSync
   - [4] if we use a local executor, we need to have the DAGs mount if we use either DAGs persistence or gitSync, no matter if DAG processor is disabled or enabled
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1166149411

   Finally green!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1130794786

   Nice! @mhenc -> maybe you want to take a look too?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1175554752

   @jedcunningham I added some combinations of DAG Processor and executors in https://github.com/apache/airflow/pull/23711/commits/a77d166a21fbdb651a98c9c45398cb0baa40332f
   
   I assumed that gitsync and persistence was always enabled to make the assertions easier to understand.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1156746743

   @dstandish @jedcunningham - I thnk it looks good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1171551764

   @jedcunningham I can set up some local tests for that. Or even better: is Airflow integration tests cover some of these scenarios? If yes, we can have some scenarios consider this feature enabled.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jedcunningham commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1147748575

   Yes, exactly. Sorry, that was a half baked comment, but that's what I was envisioning - expanding $local to cover both LocalExecutor and LocalKubernetesExecutor and using it to determine whether to mount DAGs in addition to the standalone flag.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1133883891

   Would love to get some feedback from our Helm people :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r914256938


##########
tests/charts/test_basic_helm_chart.py:
##########
@@ -113,6 +113,84 @@ def test_basic_deployments(self, version):
                 "TEST-LABEL"
             ), f"Missing label TEST-LABEL on {k8s_name}. Current labels: {labels}"
 
+    @parameterized.expand(["2.3.2", "2.4.0", "default"])
+    def test_basic_deployment_with_standalone_scheduler(self, version):

Review Comment:
   ```suggestion
       def test_basic_deployment_with_standalone_dag_processor(self, version):
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r873084230


##########
chart/templates/dagprocessor/dag-processor-serviceaccount.yaml:
##########
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+################################
+## Airflow Triggerer ServiceAccount

Review Comment:
   ```suggestion
   ## Airflow Dag Processor ServiceAccount
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] skyboi1233 commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
skyboi1233 commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1128215964

   My. Account is not secure and changes are being mad with out my promises to my accounts any way we can fix this issues 
   
   
   Sent from Yahoo Mail for iPhone
   
   
   On Monday, May 16, 2022, 5:42 PM, Jed Cunningham ***@***.***> wrote:
   
   
   
   
   @jedcunningham commented on this pull request.
   
   We no longer need the DAGs mounted in the scheduler if this is enabled, right?
   
   In chart/templates/dagprocessor/dag-processor-deployment.yaml:
   > +## Airflow Dag Processor Deployment
   +#################################
   +{{- if semverCompare ">=2.3.0" .Values.airflowVersion }}
   +{{- if .Values.dagProcessor.enabled }}
   +{{- $nodeSelector := or .Values.dagProcessor.nodeSelector .Values.nodeSelector }}
   +{{- $affinity := or .Values.dagProcessor.affinity .Values.affinity }}
   +{{- $tolerations := or .Values.dagProcessor.tolerations .Values.tolerations }}
   +{{- $topologySpreadConstraints := or .Values.dagProcessor.topologySpreadConstraints .Values.topologySpreadConstraints }}
   +{{- $securityContext := include "airflowSecurityContext" (list . .Values.dagProcessor) }}
   +kind: Deployment
   +apiVersion: apps/v1
   +metadata:
   +  name: {{ .Release.Name }}-dag-processor
   +  labels:
   +    tier: airflow
   +    component: dag-processor
   
   That second one sounds good to me.
   
   In chart/templates/_helpers.yaml:
   > @@ -653,6 +664,14 @@ Create the name of the cleanup service account to use
        airflow jobs check --job-type TriggererJob --hostname $(hostname)
    {{- end }}
    
   +{{define "dag_processor_liveness_check_command"}}
   +  - sh
   +  - -c
   +  - |
   +    CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
   +    airflow jobs check --job-type SchedulerJob --hostname $(hostname)
   
   In the meantime, you can probably leave off the job type and it'll still work.
   
   In tests/charts/test_basic_helm_chart.py:
   > @@ -97,6 +97,78 @@ def test_basic_deployments(self):
                    "TEST-LABEL"
                ), f"Missing label TEST-LABEL on {k8s_name}. Current labels: {labels}"
    
   +    def test_basic_deployment_with_standalone_scheduler(self):
   +        k8s_objects = render_chart(
   +            "TEST-BASIC",
   +            values={
   +                "chart": {
   +                    'metadata': 'AA',
   +                },
   +                'labels': {"TEST-LABEL": "TEST-VALUE"},
   +                "fullnameOverride": "TEST-BASIC",
   +                "dagProcessor": {'enabled': True},
   
   I think being disabled by default at this point makes sense.
   
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jedcunningham commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1145004608

   @gmsantos, sorry for the delay, just got around to testing this. I overlooked the fact that with LocalExecutor, the scheduler is the worker 🤦‍♂️. So the condition to mount DAGs needs to take that into account as well. You can probably use `$local` for that purpose.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1153625933

   Rebased aftere fixing the flaky test. @mhenc @jedcunningham @WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jtvmatos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
jtvmatos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1251263257

   > Different subdirs are only available as of Airlfow 2.4.0 and this is not available yet in the Helm Chart. But feel free to add it as PR.
   
   I'm not completely sure how is the best way to implement this feature, but at least I'll start by opening the issue ->  https://github.com/apache/airflow/issues/26494


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r873084559


##########
tests/charts/test_basic_helm_chart.py:
##########
@@ -97,6 +97,78 @@ def test_basic_deployments(self):
                 "TEST-LABEL"
             ), f"Missing label TEST-LABEL on {k8s_name}. Current labels: {labels}"
 
+    def test_basic_deployment_with_standalone_scheduler(self):
+        k8s_objects = render_chart(
+            "TEST-BASIC",
+            values={
+                "chart": {
+                    'metadata': 'AA',
+                },
+                'labels': {"TEST-LABEL": "TEST-VALUE"},
+                "fullnameOverride": "TEST-BASIC",
+                "dagProcessor": {'enabled': True},

Review Comment:
   If we keep it enabled by default, I can merge the assertions into the test_basic_deployment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r874774863


##########
chart/templates/_helpers.yaml:
##########
@@ -653,6 +664,14 @@ Create the name of the cleanup service account to use
     airflow jobs check --job-type TriggererJob --hostname $(hostname)
 {{- end }}
 
+{{define "dag_processor_liveness_check_command"}}
+  - sh
+  - -c
+  - |
+    CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
+    airflow jobs check --job-type SchedulerJob --hostname $(hostname)

Review Comment:
   Makes sense



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1135523236

   Related https://github.com/apache/airflow/pull/23872
   
   We might need this patch before running the standalone dag processor with the official image.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jedcunningham commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1176694779

   Thanks @gmsantos 🎉🍺


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1172131764

   > @jedcunningham I can set up some local tests for that. Or even better: is Airflow integration tests cover some of these scenarios? If yes, we can have some scenarios consider this feature enabled.
   
   Yeah I am also going to review and likely simplify the tests we already have for "K8S" scenario (those using kind) as they run far too long and IMHO they are far too repeating themselves without making good use of common setup time (but this is because they were based on our old "bash" breeze scripts and it was hard to be flexible there. When I get to rewrite those using the new Python base breeze this will be an opportunity to a) slim them down b) add some more useful scenarios once we get the tests reuse more of a common setup - so that it will not add too much build time.
   
   So for now - unit tests with helm are the way to go (though they will need some speedup as well eventually as I have a feeling the way we run them can be sped up


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r873084202


##########
chart/templates/dagprocessor/dag-processor-deployment.yaml:
##########
@@ -0,0 +1,214 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+################################
+## Airflow Dag Processor Deployment
+#################################
+{{- if semverCompare ">=2.3.0" .Values.airflowVersion }}
+{{- if .Values.dagProcessor.enabled }}
+{{- $nodeSelector := or .Values.dagProcessor.nodeSelector .Values.nodeSelector }}
+{{- $affinity := or .Values.dagProcessor.affinity .Values.affinity }}
+{{- $tolerations := or .Values.dagProcessor.tolerations .Values.tolerations }}
+{{- $topologySpreadConstraints := or .Values.dagProcessor.topologySpreadConstraints .Values.topologySpreadConstraints }}
+{{- $securityContext := include "airflowSecurityContext" (list . .Values.dagProcessor) }}
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: {{ .Release.Name }}-dag-processor
+  labels:
+    tier: airflow
+    component: dag-processor

Review Comment:
   This is the first component in templates with a composed name. How should it be referenced?
   
   At this moment, I'm following the pattern:
   
   `dagprocessor` as the path name inside the template folder
   `dag-processor` in labels and in the k8s object name
   `dagProcessor` in the values file



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r873084453


##########
chart/templates/_helpers.yaml:
##########
@@ -653,6 +664,14 @@ Create the name of the cleanup service account to use
     airflow jobs check --job-type TriggererJob --hostname $(hostname)
 {{- end }}
 
+{{define "dag_processor_liveness_check_command"}}
+  - sh
+  - -c
+  - |
+    CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
+    airflow jobs check --job-type SchedulerJob --hostname $(hostname)

Review Comment:
   We don't have a dedicated jobs check command for standalone dag processor for now. I can create in this same PR or in a separate one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r873084314


##########
chart/values.schema.json:
##########
@@ -2101,6 +2102,264 @@
                 }
             }
         },
+        "dagProcessor": {

Review Comment:
   This block is based on the triggerer component.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1128976205

   > We no longer need the DAGs mounted in the scheduler if this is enabled, right?
   
   True, that makes sense, I can remove the persistent DAG volume depending on `dagProcessor.enabled`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1129290893

   @jedcunningham rebased and incorporated the review suggestions.
   
   It's working as expected in a dev deployment:
   
   ```shell
   > cat values.yaml
   [...]
   
   dagProcessor:
     enabled: true
   
   [...]
   
   > kubectl get pods
   NAME                                                    READY   STATUS      RESTARTS   AGE
   airflow-cleanup-27546945-55mjl                          0/1     Completed   0          41m
   airflow-cleanup-27546960-mslp7                          0/1     Completed   0          26m
   airflow-cleanup-27546975-zvh75                          0/1     Completed   0          11m
   airflow-dag-processor-68479f4fb-8bfnx                   1/1     Running     0          3m52s
   airflow-scheduler-699b79bc7d-59wc2                      2/2     Running     0          3m52s
   airflow-scheduler-699b79bc7d-j7xb4                      2/2     Running     0          53s
   airflow-statsd-579db47f48-nms6x                         1/1     Running     0          29h
   airflow-triggerer-5df797bbf6-zrng6                      1/1     Running     0          3m51s
   airflow-webserver-78bcfc575-btwc4                       1/1     Running     0          3m25s
   airflow-webserver-78bcfc575-vjrv2                       1/1     Running     0          3m50s
   ```
   
   This deployment does not use gitsync or persistent volume mounted with the pods, but it should work according to the tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1145017819

   @jedcunningham okay, let me take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on a diff in pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on code in PR #23711:
URL: https://github.com/apache/airflow/pull/23711#discussion_r874055316


##########
chart/templates/dagprocessor/dag-processor-deployment.yaml:
##########
@@ -0,0 +1,214 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+################################
+## Airflow Dag Processor Deployment
+#################################
+{{- if semverCompare ">=2.3.0" .Values.airflowVersion }}
+{{- if .Values.dagProcessor.enabled }}
+{{- $nodeSelector := or .Values.dagProcessor.nodeSelector .Values.nodeSelector }}
+{{- $affinity := or .Values.dagProcessor.affinity .Values.affinity }}
+{{- $tolerations := or .Values.dagProcessor.tolerations .Values.tolerations }}
+{{- $topologySpreadConstraints := or .Values.dagProcessor.topologySpreadConstraints .Values.topologySpreadConstraints }}
+{{- $securityContext := include "airflowSecurityContext" (list . .Values.dagProcessor) }}
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: {{ .Release.Name }}-dag-processor
+  labels:
+    tier: airflow
+    component: dag-processor

Review Comment:
   A more consistent alternative would be:
   
   `dag-processor` as the path inside the template folder, labels and the k8s object name
   `dagProcessor` in the values file



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1167500445

   Hey @dstandish @jedcunningham, can you take a look at this one? It should be good to review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] gmsantos commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
gmsantos commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1147738306

   @jedcunningham I have a question about the `LocalKubernetesExecutor`, I assume it if the tasks doesn't use the kubernetes_pool, the scheduler works as a worker as well, isn't?
   
   I'm assuming that we should include `LocalKubernetesExecutor` in the $local variable as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jedcunningham commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1171565416

   The integration tests aren't that comprehensive (happy to send some links on what we do have on slack if you are curious).
   
   We should just add some coverage in the unit tests for this PR. For example, add/expand the matrix for `test_dags_gitsync_sidecar_and_init_container` and `test_no_dags_mount_or_volume_or_gitsync_sidecar_expected`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] jedcunningham merged pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
jedcunningham merged PR #23711:
URL: https://github.com/apache/airflow/pull/23711


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1251042988

   Different subdirs are only available as of Airlfow 2.4.0 and this is not available yet in the Helm Chart. But feel free to add it as PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #23711: Helmchart: Provision Standalone Dag Processor

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #23711:
URL: https://github.com/apache/airflow/pull/23711#issuecomment-1156746103

   Rebaset it to main as we fixed it there. https://github.com/apache/airflow/pull/24479 is apparently the root cause (reverted now) and we are scratching our heads with @jedcunningham why.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org