You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Shoumitra Srivastava <sh...@gmail.com> on 2017/09/25 05:44:17 UTC
Issues with postgres_operator
Hey guys,
I am currently using Airflow 1.8.2 to schedule some EMR tasks and then
execute some long running queries on our Redshift cluster. For that purpose
I am using the postgres_operator. The queries take about 30 minutes to run.
However, once they are done, the connection never closes and the operator
runs for an hour and a half more till its terminated at the 2 hour mark
every time. The logs say that the server closed the connection
unexpectedly. I've checked the logs on Redshift's end and it shows the
queries have run and the connection has been closed. Somehow, that is never
communicated back to Airflow. Any directions on what more I could check? To
give some more info, my Airflow installation is an extension of the
https://github.com/puckel/docker-airflow docker image, is run in an ECS
cluster and has SQLite as backend since I am still testing Airflow out. I
am also using Sequential Executor right now. I would appreciate any help in
this matter.
thanks,
Shoumitra
Re: Issues with postgres_operator
Posted by Shoumitra Srivastava <sh...@gmail.com>.
Hi guys,
Thank you for your quick replies. Seems like Arthur's pull request
addresses exactly what I am facing. Hopefully, it gets merged and I can
move ahead.
thanks,
Shoumitra
On Mon, Sep 25, 2017 at 8:25 AM, Arthur Purvis <ap...@lumoslabs.com>
wrote:
> fixed here: https://github.com/apache/incubator-airflow/pull/2513
>
> On Mon, Sep 25, 2017 at 10:28 AM, Driesprong, Fokko <fo...@driesprong.frl>
> wrote:
>
>> Hi Shoumitra,
>>
>> So if I understand correctly, the query finishes but the connection does
>> not close. What is the result of the query? Maybe you can also share some
>> code of how you use the PostgresOperator the communicate with Redshift.
>>
>> Regarding your config. The sequential executor is only recommended when
>> debugging, since it runs in process and only allows you to execute one
>> task
>> at the time. I would recommend upgrading to the LocalExecutor and use an
>> AWS managed Postgres for the database.
>>
>> Let me know.
>>
>> Cheers, Fokko
>>
>> 2017-09-25 7:44 GMT+02:00 Shoumitra Srivastava <sh...@gmail.com>:
>>
>> > Hey guys,
>> >
>> > I am currently using Airflow 1.8.2 to schedule some EMR tasks and then
>> > execute some long running queries on our Redshift cluster. For that
>> purpose
>> > I am using the postgres_operator. The queries take about 30 minutes to
>> run.
>> > However, once they are done, the connection never closes and the
>> operator
>> > runs for an hour and a half more till its terminated at the 2 hour mark
>> > every time. The logs say that the server closed the connection
>> > unexpectedly. I've checked the logs on Redshift's end and it shows the
>> > queries have run and the connection has been closed. Somehow, that is
>> never
>> > communicated back to Airflow. Any directions on what more I could
>> check? To
>> > give some more info, my Airflow installation is an extension of the
>> > https://github.com/puckel/docker-airflow docker image, is run in an ECS
>> > cluster and has SQLite as backend since I am still testing Airflow out.
>> I
>> > am also using Sequential Executor right now. I would appreciate any
>> help in
>> > this matter.
>> >
>> > thanks,
>> > Shoumitra
>> >
>>
>
>
Re: Issues with postgres_operator
Posted by Arthur Purvis <ap...@lumoslabs.com>.
fixed here: https://github.com/apache/incubator-airflow/pull/2513
On Mon, Sep 25, 2017 at 10:28 AM, Driesprong, Fokko <fo...@driesprong.frl>
wrote:
> Hi Shoumitra,
>
> So if I understand correctly, the query finishes but the connection does
> not close. What is the result of the query? Maybe you can also share some
> code of how you use the PostgresOperator the communicate with Redshift.
>
> Regarding your config. The sequential executor is only recommended when
> debugging, since it runs in process and only allows you to execute one task
> at the time. I would recommend upgrading to the LocalExecutor and use an
> AWS managed Postgres for the database.
>
> Let me know.
>
> Cheers, Fokko
>
> 2017-09-25 7:44 GMT+02:00 Shoumitra Srivastava <sh...@gmail.com>:
>
> > Hey guys,
> >
> > I am currently using Airflow 1.8.2 to schedule some EMR tasks and then
> > execute some long running queries on our Redshift cluster. For that
> purpose
> > I am using the postgres_operator. The queries take about 30 minutes to
> run.
> > However, once they are done, the connection never closes and the operator
> > runs for an hour and a half more till its terminated at the 2 hour mark
> > every time. The logs say that the server closed the connection
> > unexpectedly. I've checked the logs on Redshift's end and it shows the
> > queries have run and the connection has been closed. Somehow, that is
> never
> > communicated back to Airflow. Any directions on what more I could check?
> To
> > give some more info, my Airflow installation is an extension of the
> > https://github.com/puckel/docker-airflow docker image, is run in an ECS
> > cluster and has SQLite as backend since I am still testing Airflow out. I
> > am also using Sequential Executor right now. I would appreciate any help
> in
> > this matter.
> >
> > thanks,
> > Shoumitra
> >
>
Re: Issues with postgres_operator
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi Shoumitra,
So if I understand correctly, the query finishes but the connection does
not close. What is the result of the query? Maybe you can also share some
code of how you use the PostgresOperator the communicate with Redshift.
Regarding your config. The sequential executor is only recommended when
debugging, since it runs in process and only allows you to execute one task
at the time. I would recommend upgrading to the LocalExecutor and use an
AWS managed Postgres for the database.
Let me know.
Cheers, Fokko
2017-09-25 7:44 GMT+02:00 Shoumitra Srivastava <sh...@gmail.com>:
> Hey guys,
>
> I am currently using Airflow 1.8.2 to schedule some EMR tasks and then
> execute some long running queries on our Redshift cluster. For that purpose
> I am using the postgres_operator. The queries take about 30 minutes to run.
> However, once they are done, the connection never closes and the operator
> runs for an hour and a half more till its terminated at the 2 hour mark
> every time. The logs say that the server closed the connection
> unexpectedly. I've checked the logs on Redshift's end and it shows the
> queries have run and the connection has been closed. Somehow, that is never
> communicated back to Airflow. Any directions on what more I could check? To
> give some more info, my Airflow installation is an extension of the
> https://github.com/puckel/docker-airflow docker image, is run in an ECS
> cluster and has SQLite as backend since I am still testing Airflow out. I
> am also using Sequential Executor right now. I would appreciate any help in
> this matter.
>
> thanks,
> Shoumitra
>