You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/27 15:46:41 UTC

[GitHub] [airflow] barrywhart opened a new issue, #25344: Improve Airflow logging for operator Jinja template processing

barrywhart opened a new issue, #25344:
URL: https://github.com/apache/airflow/issues/25344

   ### Description
   
   When an operator uses Jinja templating, debugging issues is difficult because the Airflow task log only displays a stack trace.
   
   ### Use case/motivation
   
   When there's a templating issue, I'd like to have some specific, actionable info to help understand the problem. At minimum:
   * Which operator or task had the issue?
   * Which field had the issue?
   * What was the Jinja template?
   
   Possibly also the Jinja context, although that can be very verbose.
   
   I have prototyped this in my local Airflow dev environment, and I propose something like the following. (Note the logging commands, which are not present in the Airflow repo.)
   
   Please let me know if this sounds reasonable, and I will be happy to create a PR.
   
   def _do_render_template_fields(
       self,
       parent,
       template_fields,
       context,
       jinja_env,
       seen_oids,
   ) -> None:
       """Copied from Airflow 2.2.5 with added logging."""
       logger.info(f"BaseOperator._do_render_template_fields(): Task {self.task_id}")
       for attr_name in template_fields:
           content = getattr(parent, attr_name)
           if content:
               logger.info(f"Rendering template for '{attr_name}' field: {content!r}")
               rendered_content = self.render_template(content, context, jinja_env, seen_oids)
   +            setattr(parent, attr_name, rendered_content)
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #25344:
URL: https://github.com/apache/airflow/issues/25344#issuecomment-1197064569

   Isn't this information already available in Render tab?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25344:
URL: https://github.com/apache/airflow/issues/25344#issuecomment-1197066878

   Some people prefer CLI and logs to see problems I believe :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #25344:
URL: https://github.com/apache/airflow/issues/25344#issuecomment-1197087484

   > Some people prefer CLI and logs to see problems I believe :)
   
   For sure but Jinja may also contain connection template (secrets). If we decide to add logging then we need to consider how to handle it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25344:
URL: https://github.com/apache/airflow/issues/25344#issuecomment-1197038412

   There is one comment I want to make though - it should be rather logged "after" the fact and only when there are problems. This will complicate the PR a bit but should be relatively easy to implement. We certainly do not want to flood the logs with thousands of rendering logs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25344:
URL: https://github.com/apache/airflow/issues/25344#issuecomment-1197193943

   Logs are redacted - this should be no problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25344:
URL: https://github.com/apache/airflow/issues/25344#issuecomment-1197029085

   By all means if you can improve it - feel free. This is a huge pain sometimes !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25344: Improve Airflow logging for operator Jinja template processing
URL: https://github.com/apache/airflow/issues/25344


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] barrywhart commented on issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
barrywhart commented on issue #25344:
URL: https://github.com/apache/airflow/issues/25344#issuecomment-1201704457

   PR created!
   
   #25452


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] barrywhart commented on issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
barrywhart commented on issue #25344:
URL: https://github.com/apache/airflow/issues/25344#issuecomment-1198342819

   Thanks, I'll try and create a PR some time in the next 1-2 weeks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25344:
URL: https://github.com/apache/airflow/issues/25344#issuecomment-1197032658

   Best to iterate on a PR - then you can see the code /tests everything else and comment directly there. So feel free to open a PR about it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #25344: Improve Airflow logging for operator Jinja template processing

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25344:
URL: https://github.com/apache/airflow/issues/25344#issuecomment-1196933037

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org