Questions regarding installing Airflow provider packages

[Alex Begg <al...@gmail.com>]

Hello,

I would like to make my use of Airflow provider packages more efficient and
"leaner" in my Docker images. I am currently using them by just listing
them in `requirements.txt`, but I have a couple of questions:

1. Is it possible to trim down the number of dependencies a provider
installs? For example `apache-airflow-providers-google` has many
dependencies (like almost 30?!?), but what if all I need is one hook,
`GoogleAnalyticsHook`, which just needs like 3 or 4 dependencies? I would
like to still use that Hook without also needing the dependencies for all
the other Hooks.

2. Also, what is the difference between installing providers one by one,
like `apache-airflow-providers-google` and
`apache-airflow-providers-amazon`, and using cross-provider dependencies
such as `apache-airflow-providers-google[amazon]`?

Thanks,
Alex Begg

Re: Questions regarding installing Airflow provider packages

[Jarek Potiuk <ja...@potiuk.com>]

If you think you can chime in and help -
https://github.com/apache/airflow/issues/15933 - the issue about it.

This is a big effort and also it needs quite some effort to do such a
split. I hope someday we will do it.

J.

On Sat, Feb 26, 2022 at 12:24 AM Alex Begg <al...@gmail.com> wrote:
>
> Daniel, I looked into that, It might be a way to go, if I don't need to use any other part of the provider.
>
>
> On Fri, Feb 25, 2022 at 3:19 PM Daniel Standish <dp...@gmail.com> wrote:
>>
>> Maybe for your use case you could try installing the provider with `--no-dependencies` and then manually pip installed the deps that are needed for just the things you are using.

Re: Questions regarding installing Airflow provider packages

[Alex Begg <al...@gmail.com>]

Daniel, I looked into that, It might be a way to go, if I don't need to use
any other part of the provider.


On Fri, Feb 25, 2022 at 3:19 PM Daniel Standish <dp...@gmail.com>
wrote:

> Maybe for your use case you could try installing the provider with
> `--no-dependencies` and then manually pip installed the deps that are
> needed for just the things you are using.
>

Re: Questions regarding installing Airflow provider packages

[Daniel Standish <dp...@gmail.com>]

Maybe for your use case you could try installing the provider with
`--no-dependencies` and then manually pip installed the deps that are
needed for just the things you are using.

Re: Questions regarding installing Airflow provider packages

[Alex Begg <al...@gmail.com>]

Thanks for your help. I was afraid that was the case.

This might be better suited to discuss on the dev list, but I wonder if
eventually there can be "bare" packages made for these big cloud provider
packages that allow the use of "extras_require" in setup.py to specify
which hooks' dependencies to install. For example,
`apache-airflow-providers-google[analytics]` will install just the
dependencies for the GoogleAnalyticsHook.

In the meantime I'll just work without the provider

Alex


On Fri, Feb 25, 2022 at 3:00 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> 1. Not possible. Google is huge provider. It was considered to be
> split - but that's a significant effort
> 2, No difference, really.
>
> On Fri, Feb 25, 2022 at 11:49 PM Alex Begg <al...@gmail.com> wrote:
> >
> > Hello,
> >
> > I would like to make my use of Airflow provider packages more efficient
> and "leaner" in my Docker images. I am currently using them by just listing
> them in `requirements.txt`, but I have a couple of questions:
> >
> > 1. Is it possible to trim down the number of dependencies a provider
> installs? For example `apache-airflow-providers-google` has many
> dependencies (like almost 30?!?), but what if all I need is one hook,
> `GoogleAnalyticsHook`, which just needs like 3 or 4 dependencies? I would
> like to still use that Hook without also needing the dependencies for all
> the other Hooks.
> >
> > 2. Also, what is the difference between installing providers one by one,
> like `apache-airflow-providers-google` and
> `apache-airflow-providers-amazon`, and using cross-provider dependencies
> such as `apache-airflow-providers-google[amazon]`?
> >
> > Thanks,
> > Alex Begg
>

Re: Questions regarding installing Airflow provider packages

[Jarek Potiuk <ja...@potiuk.com>]

1. Not possible. Google is huge provider. It was considered to be
split - but that's a significant effort
2, No difference, really.

On Fri, Feb 25, 2022 at 11:49 PM Alex Begg <al...@gmail.com> wrote:
>
> Hello,
>
> I would like to make my use of Airflow provider packages more efficient and "leaner" in my Docker images. I am currently using them by just listing them in `requirements.txt`, but I have a couple of questions:
>
> 1. Is it possible to trim down the number of dependencies a provider installs? For example `apache-airflow-providers-google` has many dependencies (like almost 30?!?), but what if all I need is one hook, `GoogleAnalyticsHook`, which just needs like 3 or 4 dependencies? I would like to still use that Hook without also needing the dependencies for all the other Hooks.
>
> 2. Also, what is the difference between installing providers one by one, like `apache-airflow-providers-google` and `apache-airflow-providers-amazon`, and using cross-provider dependencies such as `apache-airflow-providers-google[amazon]`?
>
> Thanks,
> Alex Begg