Getting logs when remote logging is enabled

[Akash Sharma <2a...@gmail.com>]

Hello everyone,

Whenever remote logging is enabled, logs are only uploaded to the target
path once the tasks have been completed. This makes it harder to monitor
tasks that are long-running since there is no means to getting the logs.

I was working on a Handler that saves the chunked logs where chunking is
decided based on two factors -

   1. Max time has elapsed since the last chunking was done
   2. Max bytes have arrived since the last chunking was done

So a chunk will be saved either when the max time has elapsed or the file
size limit has been surpassed. The chunked files can then be uploaded
whenever they are created and served by stitching them back together.

Do let me know your thoughts.

Best regards,
Akash

Re: Getting logs when remote logging is enabled

[Amogh Desai <am...@gmail.com>]

Hi Akash,

Thanks for starting the discussion.

Adding a bit to what Jens said here, a lot of the logging handler might
change for
AF 3. Although what you are looking at seems very similar to what fluentbit
/ fluentd offers,
have you explored that?

FluentBit: https://fluentbit.io/
FluentD: https://www.fluentd.org/

They also have capability for docker ecosystems (needn't have to be the
traditional K8s ecosystem).
Check here: https://www.fluentd.org/guides/recipes/docker-logging

A question about "which" component are you adding the handler in? I think
we would probably benefit from
a separate service to do this that probably shares a "common" volume with
the AF components? (Although look
out for the removal of direct DB access in AF 3)


Thanks & Regards,
Amogh Desai


On Sun, Mar 9, 2025 at 12:53 AM Akash Sharma <2a...@gmail.com> wrote:

> Hi Jens,
> The point here is that the solution should be setup agnostic i.e whether
> the tasks are being run in CeleryExecutor, K8sExecutor, or
> CeleryK8sOperator etc.. or whether the executors are reachable by the web
> server or not.
>
> Best regards,
> Akash
>
> On Sat, Mar 8, 2025 at 11:14 PM Jens Scheffler <j_scheffler@gmx.de.invalid
> >
> wrote:
>
> > Hi Akash,
> >
> > so for remote logging logs still can be sourced from the worker via the
> > web server if the endpoint hosted for this is reachable. Web server
> > attempts to source the logs from worker or local file system if not
> > found on remote. This is the standard for Celery for example.
> > Alternatively a shared log file system can be used and the webserver can
> > provide logs from there.
> >
> > In Airflow 3 (soon) there will be an enhanced way to ship logs while
> > in-flight.
> >
> > Otherwise if you hsot your workers remote and you don't get a network
> > connection from webserver to your worker, then you can take a look to
> > the new Edge Worker which also streams logs in chunks from the edge site
> > to the central location.
> >
> > If you otherwise want to contribute, helping hands are always welcome.
> > The log handler structure though will probably change in Airflow 3 soon.
> > Limitations of remote log storages for S3 / Azure Blob apply that you
> > can not append chunks.
> >
> > Jens
> >
> > On 08.03.25 16:49, Akash Sharma wrote:
> > > Hello everyone,
> > >
> > > Whenever remote logging is enabled, logs are only uploaded to the
> target
> > > path once the tasks have been completed. This makes it harder to
> monitor
> > > tasks that are long-running since there is no means to getting the
> logs.
> > >
> > > I was working on a Handler that saves the chunked logs where chunking
> is
> > > decided based on two factors -
> > >
> > >     1. Max time has elapsed since the last chunking was done
> > >     2. Max bytes have arrived since the last chunking was done
> > >
> > > So a chunk will be saved either when the max time has elapsed or the
> file
> > > size limit has been surpassed. The chunked files can then be uploaded
> > > whenever they are created and served by stitching them back together.
> > >
> > > Do let me know your thoughts.
> > >
> > > Best regards,
> > > Akash
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: Getting logs when remote logging is enabled

[Akash Sharma <2a...@gmail.com>]

Hi Jens,
The point here is that the solution should be setup agnostic i.e whether
the tasks are being run in CeleryExecutor, K8sExecutor, or
CeleryK8sOperator etc.. or whether the executors are reachable by the web
server or not.

Best regards,
Akash

On Sat, Mar 8, 2025 at 11:14 PM Jens Scheffler <j_...@gmx.de.invalid>
wrote:

> Hi Akash,
>
> so for remote logging logs still can be sourced from the worker via the
> web server if the endpoint hosted for this is reachable. Web server
> attempts to source the logs from worker or local file system if not
> found on remote. This is the standard for Celery for example.
> Alternatively a shared log file system can be used and the webserver can
> provide logs from there.
>
> In Airflow 3 (soon) there will be an enhanced way to ship logs while
> in-flight.
>
> Otherwise if you hsot your workers remote and you don't get a network
> connection from webserver to your worker, then you can take a look to
> the new Edge Worker which also streams logs in chunks from the edge site
> to the central location.
>
> If you otherwise want to contribute, helping hands are always welcome.
> The log handler structure though will probably change in Airflow 3 soon.
> Limitations of remote log storages for S3 / Azure Blob apply that you
> can not append chunks.
>
> Jens
>
> On 08.03.25 16:49, Akash Sharma wrote:
> > Hello everyone,
> >
> > Whenever remote logging is enabled, logs are only uploaded to the target
> > path once the tasks have been completed. This makes it harder to monitor
> > tasks that are long-running since there is no means to getting the logs.
> >
> > I was working on a Handler that saves the chunked logs where chunking is
> > decided based on two factors -
> >
> >     1. Max time has elapsed since the last chunking was done
> >     2. Max bytes have arrived since the last chunking was done
> >
> > So a chunk will be saved either when the max time has elapsed or the file
> > size limit has been surpassed. The chunked files can then be uploaded
> > whenever they are created and served by stitching them back together.
> >
> > Do let me know your thoughts.
> >
> > Best regards,
> > Akash
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: Getting logs when remote logging is enabled

[Jens Scheffler <j_...@gmx.de.INVALID>]

Hi Akash,

so for remote logging logs still can be sourced from the worker via the
web server if the endpoint hosted for this is reachable. Web server
attempts to source the logs from worker or local file system if not
found on remote. This is the standard for Celery for example.
Alternatively a shared log file system can be used and the webserver can
provide logs from there.

In Airflow 3 (soon) there will be an enhanced way to ship logs while
in-flight.

Otherwise if you hsot your workers remote and you don't get a network
connection from webserver to your worker, then you can take a look to
the new Edge Worker which also streams logs in chunks from the edge site
to the central location.

If you otherwise want to contribute, helping hands are always welcome.
The log handler structure though will probably change in Airflow 3 soon.
Limitations of remote log storages for S3 / Azure Blob apply that you
can not append chunks.

Jens

On 08.03.25 16:49, Akash Sharma wrote:
> Hello everyone,
>
> Whenever remote logging is enabled, logs are only uploaded to the target
> path once the tasks have been completed. This makes it harder to monitor
> tasks that are long-running since there is no means to getting the logs.
>
> I was working on a Handler that saves the chunked logs where chunking is
> decided based on two factors -
>
>     1. Max time has elapsed since the last chunking was done
>     2. Max bytes have arrived since the last chunking was done
>
> So a chunk will be saved either when the max time has elapsed or the file
> size limit has been surpassed. The chunked files can then be uploaded
> whenever they are created and served by stitching them back together.
>
> Do let me know your thoughts.
>
> Best regards,
> Akash
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org