[GitHub] [airflow] kaxil commented on a change in pull request #13310: Respect LogFormat when using ES logging with Json Format

[GitBox <gi...@apache.org>]

kaxil commented on a change in pull request #13310:
URL: https://github.com/apache/airflow/pull/13310#discussion_r548774234



##########
File path: airflow/providers/elasticsearch/log/es_task_handler.py
##########
@@ -194,12 +194,26 @@ def _read(
         # to prevent it from showing in the UI.
         def concat_logs(lines):
             log_range = (len(lines) - 1) if lines[-1].message == self.end_of_log_mark.strip() else len(lines)
-            return '\n'.join([lines[i].message for i in range(log_range)])
+            return '\n'.join([self._format_msg(lines[i]) for i in range(log_range)])
 
         message = [(host, concat_logs(hosted_log)) for host, hosted_log in logs_by_host]
 
         return message, metadata
 
+    def _format_msg(self, log_line):
+        """Format ES Record to match settings.LOG_FORMAT when used with json_format"""
+        # Using formatter._style.format makes it future proof i.e.
+        # if we change the formatter style from '%' to '{' or '$', this will still work
+        if self.json_format:
+            try:
+                # pylint: disable=protected-access
+                return self.formatter._style.format(_ESJsonLogFmt(**log_line.to_dict()))
+            except Exception:  # noqa pylint: disable=broad-except

Review comment:
       For now, I can only foresee `KeyError` here i.e. if log_format has a key that is not present in ES logs:
   https://github.com/apache/airflow/blob/69d6d0239f470ac75e23160bac63408350c1835a/airflow/config_templates/default_airflow.cfg#L289




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org