[GitHub] [airflow] hussein-awala opened a new issue, #31834: Redis task handler for logs

["hussein-awala (via GitHub)" <gi...@apache.org>]

hussein-awala opened a new issue, #31834:
URL: https://github.com/apache/airflow/issues/31834

   ### Discussed in https://github.com/apache/airflow/discussions/31832
   
   <div type='discussions-op-text'>
   
   <sup>Originally posted by **michalc** June 10, 2023</sup>
   Should something like the below be in the codebase? It's a simple handler for storing Airflow task logs in Redis, enforcing a max number of entries per try, and an expiry time for the logs
   
   Happy to raise a PR (and I guessed a lot at how things should be... so suspect can be improved upon...)
   
   ```python
   class RedisHandler(logging.Handler):
       def __init__(self, client, key):
           super().__init__()
           self.client = client
           self.key = key
   
       def emit(self, record):
           p = self.client.pipeline()
           p.rpush(self.key, self.format(record))
           p.ltrim(self.key, start=-10000, end=-1)
           p.expire(self.key, time=60 * 60 * 24 * 28)
           p.execute()
   
   
   class RedisTaskHandler(FileTaskHandler, LoggingMixin):
       """
       RedisTaskHandler is a python log handler that handles and reads
       task instance logs. It extends airflow FileTaskHandler and
       uploads to and reads from Redis.
       """
   
       trigger_should_wrap = True
   
       def __init__(self, base_log_folder: str, redis_url):
           super().__init__(base_log_folder)
           self.handler = None
           self.client = redis.Redis.from_url(redis_url)
   
       def _read(
           self,
           ti,
           try_number,
           metadata=None,
       ):
           log_str = b"\n".join(
               self.client.lrange(self._render_filename(ti, try_number), start=0, end=-1)
           ).decode("utf-8")
           return log_str, {"end_of_log": True}
   
       def set_context(self, ti):
           super().set_context(ti)
           self.handler = RedisHandler(
               self.client, self._render_filename(ti, ti.try_number)
           )
           self.handler.setFormatter(self.formatter)
   ```</div>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #31834: Redis task handler for logs

["eladkal (via GitHub)" <gi...@apache.org>]

eladkal closed issue #31834: Redis task handler for logs
URL: https://github.com/apache/airflow/issues/31834


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] hussein-awala commented on issue #31834: Redis task handler for logs

["hussein-awala (via GitHub)" <gi...@apache.org>]

hussein-awala commented on issue #31834:
URL: https://github.com/apache/airflow/issues/31834#issuecomment-1585749004

   @michalc I created this issue for the discussed feature


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org