You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Veeranagouda Mukkanagoudar <mu...@gmail.com> on 2017/10/14 19:11:17 UTC
Redshift operation examples
I am new to Airflow,
Can anyone point me to Redshift/Postgress operator or task implementation
examples.
-Thanks
Veera
Re: Redshift operation examples
Posted by Veeranagouda Mukkanagoudar <mu...@gmail.com>.
Thanks Andy, I think this is helpful .
On Sat, Oct 14, 2017 at 12:22 PM, Andy Hadjigeorgiou <an...@gmail.com>
wrote:
> Hello,
>
> If you are looking for querying Redshift clusters, PostgresOperators and
> PostgresHook is what you are looking for. Here's the docs
> <https://pythonhosted.org/airflow/code.html#airflow.hooks.PostgresHook>
> for
> both. If you are looking to manage Redshift clusters, right now you'd have
> to use boto3 by hand (there is not currently a hook for managing Redshift
> clusters). I'm working on a RedshiftHook based on this ticket
> <https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-1695>, but
> it's awaiting PR#2532
> <https://github.com/apache/incubator-airflow/pull/2532> (moving s3 hook to
> boto3).
>
> Hope that is helpful!
>
> Best,
>
> Andy
>
> -------
> Software Engineer | Fundera
>
> On Sat, Oct 14, 2017 at 3:11 PM, Veeranagouda Mukkanagoudar <
> mukkanagoudar@gmail.com> wrote:
>
> > I am new to Airflow,
> > Can anyone point me to Redshift/Postgress operator or task implementation
> > examples.
> >
> > -Thanks
> > Veera
> >
>
Re: Redshift operation examples
Posted by Andy Hadjigeorgiou <an...@gmail.com>.
This blog post
<http://tech.marksblogg.com/airflow-postgres-redis-forex.html> has a good
example of using PostgresHook to query a database - all you'd need to
change is the connection info (to however you'd normally access your
Redshift cluster to query).
Andy
-------
Software Engineer | Fundera
On Sat, Oct 14, 2017 at 3:22 PM, Andy Hadjigeorgiou <an...@gmail.com>
wrote:
> Hello,
>
> If you are looking for querying Redshift clusters, PostgresOperators and
> PostgresHook is what you are looking for. Here's the docs
> <https://pythonhosted.org/airflow/code.html#airflow.hooks.PostgresHook> for
> both. If you are looking to manage Redshift clusters, right now you'd have
> to use boto3 by hand (there is not currently a hook for managing Redshift
> clusters). I'm working on a RedshiftHook based on this ticket
> <https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-1695>,
> but it's awaiting PR#2532
> <https://github.com/apache/incubator-airflow/pull/2532> (moving s3 hook
> to boto3).
>
> Hope that is helpful!
>
> Best,
>
> Andy
>
> -------
> Software Engineer | Fundera
>
> On Sat, Oct 14, 2017 at 3:11 PM, Veeranagouda Mukkanagoudar <
> mukkanagoudar@gmail.com> wrote:
>
>> I am new to Airflow,
>> Can anyone point me to Redshift/Postgress operator or task implementation
>> examples.
>>
>> -Thanks
>> Veera
>>
>
>
Re: Redshift operation examples
Posted by Andy Hadjigeorgiou <an...@gmail.com>.
Hello,
If you are looking for querying Redshift clusters, PostgresOperators and
PostgresHook is what you are looking for. Here's the docs
<https://pythonhosted.org/airflow/code.html#airflow.hooks.PostgresHook> for
both. If you are looking to manage Redshift clusters, right now you'd have
to use boto3 by hand (there is not currently a hook for managing Redshift
clusters). I'm working on a RedshiftHook based on this ticket
<https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-1695>, but
it's awaiting PR#2532
<https://github.com/apache/incubator-airflow/pull/2532> (moving s3 hook to
boto3).
Hope that is helpful!
Best,
Andy
-------
Software Engineer | Fundera
On Sat, Oct 14, 2017 at 3:11 PM, Veeranagouda Mukkanagoudar <
mukkanagoudar@gmail.com> wrote:
> I am new to Airflow,
> Can anyone point me to Redshift/Postgress operator or task implementation
> examples.
>
> -Thanks
> Veera
>