[GitHub] [airflow] jtvmatos commented on issue #26494: [Helm Chart] Support multiple DagProcessors

[GitBox <gi...@apache.org>]

jtvmatos commented on issue #26494:
URL: https://github.com/apache/airflow/issues/26494#issuecomment-1253268272

   Hi @potiuk thanks for the feedback.
   
   Unfortunately in the coming weeks I won't be available to create this PR (I will be unavailable for more than a month... if no one solves it by then, I can open the PR....)
   
   But yes, this PR can be tricky and can create a breaking change in the current processor configurations. For example, creating a nested level for multiple DagProcessors could be an option:
   ```
   # Airflow Dag Processor Config
   dagProcessor:
     enabled: false
       # Airflow dag processors in the deployment
     processor:
       # Processor 1
       - name: processor-1 # Alias for metadata.name
         replicas: 1
         revisionHistoryLimit: ~
         command: ~
         args: ["bash", "-c", "exec airflow dag-processor"]
         strategy:
        (...)
       # Processor 2
       # - name: processor-2
       #   args: ["bash", "-c", "exec airflow dag-processor --subdir  ..."]
   ```
   With this values.yaml, will be possible to set the processors configurations as currently used for `env` or `secrets`: `dagProcessor.processor[0]...`,  `dagProcessor.processor[1]...`  for N processors (>=2.4.0).
   
   Alternatively, we can just add a `extraDagProcessor` list:
   ```
   # Airflow Dag Processor Config
   dagProcessor:
     enabled: false
      (...)
     env: []
   
     extraDagProcessor: []
   ``` 
   This list is not `breaking` for the current settings. However, it may imply that there is a "master" processor.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org