You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Sandeep S <30...@gmail.com> on 2020/02/08 00:47:05 UTC
Tidal-Airflow Intrgration
Hi,
We are moving from Tidal to Airflow for all ETL applications in our
organization. Some of our ETL Airflow jobs are still dependent on Tidal
jobs. Is there a way we can have Tidal-Airflow integration to read the
Tidal job status and invoke Airflow jobs.
Regards
Sandeep
Re: Tidal-Airflow Intrgration
Posted by Kamil Breguła <ka...@polidea.com>.
I am afraid that migration from Tidal to Airflow is not very common
and a ready solution may not exist. Even if migrations take place, the
code responsible for this is public. In large organizations, getting
permission to share the code is a lot of work. I think it will be
easier and faster to develop some custom mechanisms by yourself and
then share this code in the community.
On Sat, Feb 8, 2020 at 7:37 PM Sandeep S <30...@gmail.com> wrote:
>
> Tidal has an api but I couldnt find much details on getting the job status. Has anyone tried this Tidal-Airflow integration in their organization.
>
> Appreciate any help.
>
> Regards
> Sandeep
>
> On Fri, Feb 7, 2020 at 8:03 PM Kamil Breguła <ka...@polidea.com> wrote:
>>
>> Does Tidal have an API that allows you to read the necessary data? If
>> so, you can write custom operators. Here is a guide:
>> https://airflow.readthedocs.io/en/latest/howto/custom-operator.html
>>
>> Here is a list of operators and hooks:
>> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html
>>
>> Best regards,
>> Kamil
>>
>> On Sat, Feb 8, 2020 at 1:47 AM Sandeep S <30...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > We are moving from Tidal to Airflow for all ETL applications in our organization. Some of our ETL Airflow jobs are still dependent on Tidal jobs. Is there a way we can have Tidal-Airflow integration to read the Tidal job status and invoke Airflow jobs.
>> >
>> > Regards
>> > Sandeep
Re: Tidal-Airflow Intrgration
Posted by Sandeep S <30...@gmail.com>.
Tidal has an api but I couldnt find much details on getting the job status.
Has anyone tried this Tidal-Airflow integration in their organization.
Appreciate any help.
Regards
Sandeep
On Fri, Feb 7, 2020 at 8:03 PM Kamil Breguła <ka...@polidea.com>
wrote:
> Does Tidal have an API that allows you to read the necessary data? If
> so, you can write custom operators. Here is a guide:
> https://airflow.readthedocs.io/en/latest/howto/custom-operator.html
>
> Here is a list of operators and hooks:
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html
>
> Best regards,
> Kamil
>
> On Sat, Feb 8, 2020 at 1:47 AM Sandeep S <30...@gmail.com> wrote:
> >
> > Hi,
> >
> > We are moving from Tidal to Airflow for all ETL applications in our
> organization. Some of our ETL Airflow jobs are still dependent on Tidal
> jobs. Is there a way we can have Tidal-Airflow integration to read the
> Tidal job status and invoke Airflow jobs.
> >
> > Regards
> > Sandeep
>
Re: Tidal-Airflow Intrgration
Posted by Sandeep S <30...@gmail.com>.
Tidal has an api but I couldnt find much details on getting the job status.
Has anyone tried this Tidal-Airflow integration in their organization.
Appreciate any help.
Regards
Sandeep
On Fri, Feb 7, 2020 at 8:03 PM Kamil Breguła <ka...@polidea.com>
wrote:
> Does Tidal have an API that allows you to read the necessary data? If
> so, you can write custom operators. Here is a guide:
> https://airflow.readthedocs.io/en/latest/howto/custom-operator.html
>
> Here is a list of operators and hooks:
> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html
>
> Best regards,
> Kamil
>
> On Sat, Feb 8, 2020 at 1:47 AM Sandeep S <30...@gmail.com> wrote:
> >
> > Hi,
> >
> > We are moving from Tidal to Airflow for all ETL applications in our
> organization. Some of our ETL Airflow jobs are still dependent on Tidal
> jobs. Is there a way we can have Tidal-Airflow integration to read the
> Tidal job status and invoke Airflow jobs.
> >
> > Regards
> > Sandeep
>
Re: Tidal-Airflow Intrgration
Posted by Kamil Breguła <ka...@polidea.com>.
Does Tidal have an API that allows you to read the necessary data? If
so, you can write custom operators. Here is a guide:
https://airflow.readthedocs.io/en/latest/howto/custom-operator.html
Here is a list of operators and hooks:
https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html
Best regards,
Kamil
On Sat, Feb 8, 2020 at 1:47 AM Sandeep S <30...@gmail.com> wrote:
>
> Hi,
>
> We are moving from Tidal to Airflow for all ETL applications in our organization. Some of our ETL Airflow jobs are still dependent on Tidal jobs. Is there a way we can have Tidal-Airflow integration to read the Tidal job status and invoke Airflow jobs.
>
> Regards
> Sandeep
Re: Tidal-Airflow Intrgration
Posted by Kamil Breguła <ka...@polidea.com>.
Does Tidal have an API that allows you to read the necessary data? If
so, you can write custom operators. Here is a guide:
https://airflow.readthedocs.io/en/latest/howto/custom-operator.html
Here is a list of operators and hooks:
https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html
Best regards,
Kamil
On Sat, Feb 8, 2020 at 1:47 AM Sandeep S <30...@gmail.com> wrote:
>
> Hi,
>
> We are moving from Tidal to Airflow for all ETL applications in our organization. Some of our ETL Airflow jobs are still dependent on Tidal jobs. Is there a way we can have Tidal-Airflow integration to read the Tidal job status and invoke Airflow jobs.
>
> Regards
> Sandeep