You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jawahar Panchal <jp...@hawk.iit.edu> on 2017/07/16 07:56:19 UTC
AIRFLOW-1258
Hi!
I am currently running a couple of long-running tasks on a database/dataset at school for a project that results in behavior/log output similar to what was flagged in this bug: https://issues.apache.org/jira/browse/AIRFLOW-1258 <https://issues.apache.org/jira/browse/AIRFLOW-1258>
Wasn’t sure if anyone on the list had seen anything similar, or would know what I can do to possibly debug further/patch. As it takes 1hr to test a change, needless to say any pointers from the dev team on the right direction to look within the codebase would be much appreciated! :)
Thanks in advance for everyone’s/anyone's time and help - am not an Airflow expert, but am hopefully learning quickly enough to help resolve this issue (if I am ‘barking up the right tree’ with this bug number…)
Cheers,
J
Re: AIRFLOW-1258
Posted by Bolke de Bruin <bd...@gmail.com>.
Not sure if it can make it into 1.8.2 -> as it has been voted upon successfully. It hasn’t been officially out yet as we need to fix some stuff for the IPMC, but that is just voting upon another tarball. Normally we can do a 1.8.3 pretty quickly though.
Bolke
> On 16 Jul 2017, at 17:51, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>
> Apologies for the self-reply - however, was digging in a bit more to the visibility timeout - if that is indeed the case, then I had some more info/questions:
>
> - Is there a way to patch this on our own build via the following on 1.8.1 - I see the following pull request/commits as a way to pass in BROKER_TRANSPORT_OPTIONS to set visibility_timeout: https://github.com/apache/incubator-airflow/pull/2143 <https://github.com/apache/incubator-airflow/pull/2143> which points to the following PR: https://issues.apache.org/jira/browse/AIRFLOW-966 <https://issues.apache.org/jira/browse/AIRFLOW-966>
>
> - Is there a way to get a fix for this issue (with the above patches or otherwise if it is a trivial patch to allow for passing visibility_timeout to the celery workers/redis) into 1.8.2 before release?! :)
>
> Cheers,
> J
>
>
>> On Jul 16, 2017, at 10:36 AM, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>>
>> Awesome - thank you for the help. Does this have anything to do with the default 1h visibility timeout when using redis as a broker? Have been doing some more digging, and was not aware of this caveat - however, am not sure if this behavior is what leads to the symptoms I am seeing (as well as what is documented in the bug report)…
>>
>> Thanks again for the help - much appreciated!
>>
>> Cheers,
>> J
>>
>>> On Jul 16, 2017, at 3:22 AM, Alex Guziel <al...@airbnb.com.INVALID> wrote:
>>>
>>> I think this may be related to a celery bug. I'll follow up with more
>>> details later.
>>>
>>> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jp...@hawk.iit.edu>
>>> wrote:
>>>
>>>> Hi!
>>>>
>>>> I am currently running a couple of long-running tasks on a
>>>> database/dataset at school for a project that results in behavior/log
>>>> output similar to what was flagged in this bug:
>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>>>
>>>> Wasn’t sure if anyone on the list had seen anything similar, or would know
>>>> what I can do to possibly debug further/patch. As it takes 1hr to test a
>>>> change, needless to say any pointers from the dev team on the right
>>>> direction to look within the codebase would be much appreciated! :)
>>>>
>>>> Thanks in advance for everyone’s/anyone's time and help - am not an
>>>> Airflow expert, but am hopefully learning quickly enough to help resolve
>>>> this issue (if I am ‘barking up the right tree’ with this bug number…)
>>>>
>>>> Cheers,
>>>> J
>>>>
>>>>
>>
>
Re: AIRFLOW-1258
Posted by Jawahar Panchal <jp...@hawk.iit.edu>.
Apologies for the self-reply - however, was digging in a bit more to the visibility timeout - if that is indeed the case, then I had some more info/questions:
- Is there a way to patch this on our own build via the following on 1.8.1 - I see the following pull request/commits as a way to pass in BROKER_TRANSPORT_OPTIONS to set visibility_timeout: https://github.com/apache/incubator-airflow/pull/2143 <https://github.com/apache/incubator-airflow/pull/2143> which points to the following PR: https://issues.apache.org/jira/browse/AIRFLOW-966 <https://issues.apache.org/jira/browse/AIRFLOW-966>
- Is there a way to get a fix for this issue (with the above patches or otherwise if it is a trivial patch to allow for passing visibility_timeout to the celery workers/redis) into 1.8.2 before release?! :)
Cheers,
J
> On Jul 16, 2017, at 10:36 AM, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>
> Awesome - thank you for the help. Does this have anything to do with the default 1h visibility timeout when using redis as a broker? Have been doing some more digging, and was not aware of this caveat - however, am not sure if this behavior is what leads to the symptoms I am seeing (as well as what is documented in the bug report)…
>
> Thanks again for the help - much appreciated!
>
> Cheers,
> J
>
>> On Jul 16, 2017, at 3:22 AM, Alex Guziel <al...@airbnb.com.INVALID> wrote:
>>
>> I think this may be related to a celery bug. I'll follow up with more
>> details later.
>>
>> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jp...@hawk.iit.edu>
>> wrote:
>>
>>> Hi!
>>>
>>> I am currently running a couple of long-running tasks on a
>>> database/dataset at school for a project that results in behavior/log
>>> output similar to what was flagged in this bug:
>>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
>>> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>>
>>> Wasn’t sure if anyone on the list had seen anything similar, or would know
>>> what I can do to possibly debug further/patch. As it takes 1hr to test a
>>> change, needless to say any pointers from the dev team on the right
>>> direction to look within the codebase would be much appreciated! :)
>>>
>>> Thanks in advance for everyone’s/anyone's time and help - am not an
>>> Airflow expert, but am hopefully learning quickly enough to help resolve
>>> this issue (if I am ‘barking up the right tree’ with this bug number…)
>>>
>>> Cheers,
>>> J
>>>
>>>
>
Re: AIRFLOW-1258
Posted by Jawahar Panchal <jp...@hawk.iit.edu>.
Awesome - thank you for the help. Does this have anything to do with the default 1h visibility timeout when using redis as a broker? Have been doing some more digging, and was not aware of this caveat - however, am not sure if this behavior is what leads to the symptoms I am seeing (as well as what is documented in the bug report)…
Thanks again for the help - much appreciated!
Cheers,
J
> On Jul 16, 2017, at 3:22 AM, Alex Guziel <al...@airbnb.com.INVALID> wrote:
>
> I think this may be related to a celery bug. I'll follow up with more
> details later.
>
> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jp...@hawk.iit.edu>
> wrote:
>
>> Hi!
>>
>> I am currently running a couple of long-running tasks on a
>> database/dataset at school for a project that results in behavior/log
>> output similar to what was flagged in this bug:
>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
>> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>
>> Wasn’t sure if anyone on the list had seen anything similar, or would know
>> what I can do to possibly debug further/patch. As it takes 1hr to test a
>> change, needless to say any pointers from the dev team on the right
>> direction to look within the codebase would be much appreciated! :)
>>
>> Thanks in advance for everyone’s/anyone's time and help - am not an
>> Airflow expert, but am hopefully learning quickly enough to help resolve
>> this issue (if I am ‘barking up the right tree’ with this bug number…)
>>
>> Cheers,
>> J
>>
>>
Re: AIRFLOW-1258
Posted by Jawahar Panchal <jp...@hawk.iit.edu>.
> On Jul 26, 2017, at 3:07 PM, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>
>
>> On Jul 26, 2017, at 2:25 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Can you explain what is solved by the patch? It only seems to set a empty default value?
>>
>> If that solves it it also seems to be a bug with celery. Or did you set some options? That would be great to share.
>>
>
> Apologies - forgot to include that the config option that was set via BROKER_TRANSPORT_OPTIONS was the visibility_timeout which I configured to >1d in seconds. So not sure if that is something that should be set in the config template with a sane default (say 1d), or if it should just be documented and the config should be left with empty broker_transport_option json.
>
Hi again - just wanted to see if 1.8.2 was already built/ready for release to see if this is includable, or whether it will have to wait for 1.8.3?
> Cheers,
> J
>
>
>> Bolke
>>
>> Sent from my iPhone
>>
>>> On 26 Jul 2017, at 19:44, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>>>
>>> Howdy - wanted to provide an update - the following patch applied manually to a clean 1.8.1 install addressed the issue:
>>>
>>> https://github.com/apache/incubator-airflow/pull/2143 <https://github.com/apache/incubator-airflow/pull/2143>
>>>
>>> We have confirmed/verified this with jobs running over multiple hours on our instance - so the above pull request can close both of the below:
>>>
>>> https://issues.apache.org/jira/browse/AIRFLOW-966 <https://issues.apache.org/jira/browse/AIRFLOW-966>
>>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>>
>>> As I have seen further discussion on 1.8.2 with the final build not being tagged yet (iirc), would it be possibly to merge this for the 1.8.2 release? As it currently is, someone using Airflow+Celery will have a broken configuration for long-running jobs - would the dev team consider this major/critical enough to include?
>>>
>>> Cheers,
>>> J
>>>
>>>> On Jul 17, 2017, at 3:15 PM, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>>>>
>>>> Hi again!
>>>>
>>>>
>>>>> On Jul 16, 2017, at 3:22 AM, Alex Guziel <alex.guziel@airbnb.com.INVALID <ma...@airbnb.com.invalid>> wrote:
>>>>>
>>>>> I think this may be related to a celery bug. I'll follow up with more
>>>>> details later.
>>>>>
>>>>
>>>> Just replying back to the note earlier in the thread - apologies for the earlier top-posting, got a bit excited that I might have found the issue, and of course lack of sleep results in one doing terrible, terrible things… :)
>>>>
>>>> Any idea if my suspicion around the 1h default visibility timeout between celery/redis is the culprit?
>>>>
>>>>> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jp...@hawk.iit.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> I am currently running a couple of long-running tasks on a
>>>>>> database/dataset at school for a project that results in behavior/log
>>>>>> output similar to what was flagged in this bug:
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>>>>>
>>>>>> Wasn’t sure if anyone on the list had seen anything similar, or would know
>>>>>> what I can do to possibly debug further/patch. As it takes 1hr to test a
>>>>>> change, needless to say any pointers from the dev team on the right
>>>>>> direction to look within the codebase would be much appreciated! :)
>>>>>>
>>>>>> Thanks in advance for everyone’s/anyone's time and help - am not an
>>>>>> Airflow expert, but am hopefully learning quickly enough to help resolve
>>>>>> this issue (if I am ‘barking up the right tree’ with this bug number…)
>>>>>>
>>>>>> Cheers,
>>>>>> J
>>>>>>
>>>>>>
>>>>
>>>> Cheers,
>>>> J
>>>
>
Re: AIRFLOW-1258
Posted by Jawahar Panchal <jp...@hawk.iit.edu>.
Hi again to the list - now that 1.8.2 is out - would it be possible to merge this for 1.8.3?
J
> On Jul 26, 2017, at 3:07 PM, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>
>
>> On Jul 26, 2017, at 2:25 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> Can you explain what is solved by the patch? It only seems to set a empty default value?
>>
>> If that solves it it also seems to be a bug with celery. Or did you set some options? That would be great to share.
>>
>
> Apologies - forgot to include that the config option that was set via BROKER_TRANSPORT_OPTIONS was the visibility_timeout which I configured to >1d in seconds. So not sure if that is something that should be set in the config template with a sane default (say 1d), or if it should just be documented and the config should be left with empty broker_transport_option json.
>
> Cheers,
> J
>
>
>> Bolke
>>
>> Sent from my iPhone
>>
>>> On 26 Jul 2017, at 19:44, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>>>
>>> Howdy - wanted to provide an update - the following patch applied manually to a clean 1.8.1 install addressed the issue:
>>>
>>> https://github.com/apache/incubator-airflow/pull/2143 <https://github.com/apache/incubator-airflow/pull/2143>
>>>
>>> We have confirmed/verified this with jobs running over multiple hours on our instance - so the above pull request can close both of the below:
>>>
>>> https://issues.apache.org/jira/browse/AIRFLOW-966 <https://issues.apache.org/jira/browse/AIRFLOW-966>
>>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>>
>>> As I have seen further discussion on 1.8.2 with the final build not being tagged yet (iirc), would it be possibly to merge this for the 1.8.2 release? As it currently is, someone using Airflow+Celery will have a broken configuration for long-running jobs - would the dev team consider this major/critical enough to include?
>>>
>>> Cheers,
>>> J
>>>
>>>> On Jul 17, 2017, at 3:15 PM, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>>>>
>>>> Hi again!
>>>>
>>>>
>>>>> On Jul 16, 2017, at 3:22 AM, Alex Guziel <alex.guziel@airbnb.com.INVALID <ma...@airbnb.com.invalid>> wrote:
>>>>>
>>>>> I think this may be related to a celery bug. I'll follow up with more
>>>>> details later.
>>>>>
>>>>
>>>> Just replying back to the note earlier in the thread - apologies for the earlier top-posting, got a bit excited that I might have found the issue, and of course lack of sleep results in one doing terrible, terrible things… :)
>>>>
>>>> Any idea if my suspicion around the 1h default visibility timeout between celery/redis is the culprit?
>>>>
>>>>> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jp...@hawk.iit.edu>
>>>>> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> I am currently running a couple of long-running tasks on a
>>>>>> database/dataset at school for a project that results in behavior/log
>>>>>> output similar to what was flagged in this bug:
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>>>>>
>>>>>> Wasn’t sure if anyone on the list had seen anything similar, or would know
>>>>>> what I can do to possibly debug further/patch. As it takes 1hr to test a
>>>>>> change, needless to say any pointers from the dev team on the right
>>>>>> direction to look within the codebase would be much appreciated! :)
>>>>>>
>>>>>> Thanks in advance for everyone’s/anyone's time and help - am not an
>>>>>> Airflow expert, but am hopefully learning quickly enough to help resolve
>>>>>> this issue (if I am ‘barking up the right tree’ with this bug number…)
>>>>>>
>>>>>> Cheers,
>>>>>> J
>>>>>>
>>>>>>
>>>>
>>>> Cheers,
>>>> J
>>>
>
Re: AIRFLOW-1258
Posted by Jawahar Panchal <jp...@hawk.iit.edu>.
> On Jul 26, 2017, at 2:25 PM, Bolke de Bruin <bd...@gmail.com> wrote:
>
> Can you explain what is solved by the patch? It only seems to set a empty default value?
>
> If that solves it it also seems to be a bug with celery. Or did you set some options? That would be great to share.
>
Apologies - forgot to include that the config option that was set via BROKER_TRANSPORT_OPTIONS was the visibility_timeout which I configured to >1d in seconds. So not sure if that is something that should be set in the config template with a sane default (say 1d), or if it should just be documented and the config should be left with empty broker_transport_option json.
Cheers,
J
> Bolke
>
> Sent from my iPhone
>
>> On 26 Jul 2017, at 19:44, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>>
>> Howdy - wanted to provide an update - the following patch applied manually to a clean 1.8.1 install addressed the issue:
>>
>> https://github.com/apache/incubator-airflow/pull/2143 <https://github.com/apache/incubator-airflow/pull/2143>
>>
>> We have confirmed/verified this with jobs running over multiple hours on our instance - so the above pull request can close both of the below:
>>
>> https://issues.apache.org/jira/browse/AIRFLOW-966 <https://issues.apache.org/jira/browse/AIRFLOW-966>
>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>
>> As I have seen further discussion on 1.8.2 with the final build not being tagged yet (iirc), would it be possibly to merge this for the 1.8.2 release? As it currently is, someone using Airflow+Celery will have a broken configuration for long-running jobs - would the dev team consider this major/critical enough to include?
>>
>> Cheers,
>> J
>>
>>> On Jul 17, 2017, at 3:15 PM, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>>>
>>> Hi again!
>>>
>>>
>>>> On Jul 16, 2017, at 3:22 AM, Alex Guziel <alex.guziel@airbnb.com.INVALID <ma...@airbnb.com.invalid>> wrote:
>>>>
>>>> I think this may be related to a celery bug. I'll follow up with more
>>>> details later.
>>>>
>>>
>>> Just replying back to the note earlier in the thread - apologies for the earlier top-posting, got a bit excited that I might have found the issue, and of course lack of sleep results in one doing terrible, terrible things… :)
>>>
>>> Any idea if my suspicion around the 1h default visibility timeout between celery/redis is the culprit?
>>>
>>>> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jp...@hawk.iit.edu>
>>>> wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> I am currently running a couple of long-running tasks on a
>>>>> database/dataset at school for a project that results in behavior/log
>>>>> output similar to what was flagged in this bug:
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>>>>
>>>>> Wasn’t sure if anyone on the list had seen anything similar, or would know
>>>>> what I can do to possibly debug further/patch. As it takes 1hr to test a
>>>>> change, needless to say any pointers from the dev team on the right
>>>>> direction to look within the codebase would be much appreciated! :)
>>>>>
>>>>> Thanks in advance for everyone’s/anyone's time and help - am not an
>>>>> Airflow expert, but am hopefully learning quickly enough to help resolve
>>>>> this issue (if I am ‘barking up the right tree’ with this bug number…)
>>>>>
>>>>> Cheers,
>>>>> J
>>>>>
>>>>>
>>>
>>> Cheers,
>>> J
>>
Re: AIRFLOW-1258
Posted by Bolke de Bruin <bd...@gmail.com>.
Can you explain what is solved by the patch? It only seems to set a empty default value?
If that solves it it also seems to be a bug with celery. Or did you set some options? That would be great to share.
Bolke
Sent from my iPhone
> On 26 Jul 2017, at 19:44, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>
> Howdy - wanted to provide an update - the following patch applied manually to a clean 1.8.1 install addressed the issue:
>
> https://github.com/apache/incubator-airflow/pull/2143 <https://github.com/apache/incubator-airflow/pull/2143>
>
> We have confirmed/verified this with jobs running over multiple hours on our instance - so the above pull request can close both of the below:
>
> https://issues.apache.org/jira/browse/AIRFLOW-966 <https://issues.apache.org/jira/browse/AIRFLOW-966>
> https://issues.apache.org/jira/browse/AIRFLOW-1258 <https://issues.apache.org/jira/browse/AIRFLOW-1258>
>
> As I have seen further discussion on 1.8.2 with the final build not being tagged yet (iirc), would it be possibly to merge this for the 1.8.2 release? As it currently is, someone using Airflow+Celery will have a broken configuration for long-running jobs - would the dev team consider this major/critical enough to include?
>
> Cheers,
> J
>
>> On Jul 17, 2017, at 3:15 PM, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>>
>> Hi again!
>>
>>
>>> On Jul 16, 2017, at 3:22 AM, Alex Guziel <alex.guziel@airbnb.com.INVALID <ma...@airbnb.com.invalid>> wrote:
>>>
>>> I think this may be related to a celery bug. I'll follow up with more
>>> details later.
>>>
>>
>> Just replying back to the note earlier in the thread - apologies for the earlier top-posting, got a bit excited that I might have found the issue, and of course lack of sleep results in one doing terrible, terrible things… :)
>>
>> Any idea if my suspicion around the 1h default visibility timeout between celery/redis is the culprit?
>>
>>> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jp...@hawk.iit.edu>
>>> wrote:
>>>
>>>> Hi!
>>>>
>>>> I am currently running a couple of long-running tasks on a
>>>> database/dataset at school for a project that results in behavior/log
>>>> output similar to what was flagged in this bug:
>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
>>>> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>>>
>>>> Wasn’t sure if anyone on the list had seen anything similar, or would know
>>>> what I can do to possibly debug further/patch. As it takes 1hr to test a
>>>> change, needless to say any pointers from the dev team on the right
>>>> direction to look within the codebase would be much appreciated! :)
>>>>
>>>> Thanks in advance for everyone’s/anyone's time and help - am not an
>>>> Airflow expert, but am hopefully learning quickly enough to help resolve
>>>> this issue (if I am ‘barking up the right tree’ with this bug number…)
>>>>
>>>> Cheers,
>>>> J
>>>>
>>>>
>>
>> Cheers,
>> J
>
Re: AIRFLOW-1258
Posted by Jawahar Panchal <jp...@hawk.iit.edu>.
Howdy - wanted to provide an update - the following patch applied manually to a clean 1.8.1 install addressed the issue:
https://github.com/apache/incubator-airflow/pull/2143 <https://github.com/apache/incubator-airflow/pull/2143>
We have confirmed/verified this with jobs running over multiple hours on our instance - so the above pull request can close both of the below:
https://issues.apache.org/jira/browse/AIRFLOW-966 <https://issues.apache.org/jira/browse/AIRFLOW-966>
https://issues.apache.org/jira/browse/AIRFLOW-1258 <https://issues.apache.org/jira/browse/AIRFLOW-1258>
As I have seen further discussion on 1.8.2 with the final build not being tagged yet (iirc), would it be possibly to merge this for the 1.8.2 release? As it currently is, someone using Airflow+Celery will have a broken configuration for long-running jobs - would the dev team consider this major/critical enough to include?
Cheers,
J
> On Jul 17, 2017, at 3:15 PM, Jawahar Panchal <jp...@hawk.iit.edu> wrote:
>
> Hi again!
>
>
>> On Jul 16, 2017, at 3:22 AM, Alex Guziel <alex.guziel@airbnb.com.INVALID <ma...@airbnb.com.invalid>> wrote:
>>
>> I think this may be related to a celery bug. I'll follow up with more
>> details later.
>>
>
> Just replying back to the note earlier in the thread - apologies for the earlier top-posting, got a bit excited that I might have found the issue, and of course lack of sleep results in one doing terrible, terrible things… :)
>
> Any idea if my suspicion around the 1h default visibility timeout between celery/redis is the culprit?
>
>> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jp...@hawk.iit.edu>
>> wrote:
>>
>>> Hi!
>>>
>>> I am currently running a couple of long-running tasks on a
>>> database/dataset at school for a project that results in behavior/log
>>> output similar to what was flagged in this bug:
>>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
>>> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>>
>>> Wasn’t sure if anyone on the list had seen anything similar, or would know
>>> what I can do to possibly debug further/patch. As it takes 1hr to test a
>>> change, needless to say any pointers from the dev team on the right
>>> direction to look within the codebase would be much appreciated! :)
>>>
>>> Thanks in advance for everyone’s/anyone's time and help - am not an
>>> Airflow expert, but am hopefully learning quickly enough to help resolve
>>> this issue (if I am ‘barking up the right tree’ with this bug number…)
>>>
>>> Cheers,
>>> J
>>>
>>>
>
> Cheers,
> J
Re: AIRFLOW-1258
Posted by Jawahar Panchal <jp...@hawk.iit.edu>.
Hi again!
> On Jul 16, 2017, at 3:22 AM, Alex Guziel <al...@airbnb.com.INVALID> wrote:
>
> I think this may be related to a celery bug. I'll follow up with more
> details later.
>
Just replying back to the note earlier in the thread - apologies for the earlier top-posting, got a bit excited that I might have found the issue, and of course lack of sleep results in one doing terrible, terrible things… :)
Any idea if my suspicion around the 1h default visibility timeout between celery/redis is the culprit?
> On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jp...@hawk.iit.edu>
> wrote:
>
>> Hi!
>>
>> I am currently running a couple of long-running tasks on a
>> database/dataset at school for a project that results in behavior/log
>> output similar to what was flagged in this bug:
>> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
>> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>>
>> Wasn’t sure if anyone on the list had seen anything similar, or would know
>> what I can do to possibly debug further/patch. As it takes 1hr to test a
>> change, needless to say any pointers from the dev team on the right
>> direction to look within the codebase would be much appreciated! :)
>>
>> Thanks in advance for everyone’s/anyone's time and help - am not an
>> Airflow expert, but am hopefully learning quickly enough to help resolve
>> this issue (if I am ‘barking up the right tree’ with this bug number…)
>>
>> Cheers,
>> J
>>
>>
Cheers,
J
Re: AIRFLOW-1258
Posted by Alex Guziel <al...@airbnb.com.INVALID>.
I think this may be related to a celery bug. I'll follow up with more
details later.
On Sun, Jul 16, 2017 at 12:56 AM Jawahar Panchal <jp...@hawk.iit.edu>
wrote:
> Hi!
>
> I am currently running a couple of long-running tasks on a
> database/dataset at school for a project that results in behavior/log
> output similar to what was flagged in this bug:
> https://issues.apache.org/jira/browse/AIRFLOW-1258 <
> https://issues.apache.org/jira/browse/AIRFLOW-1258>
>
> Wasn’t sure if anyone on the list had seen anything similar, or would know
> what I can do to possibly debug further/patch. As it takes 1hr to test a
> change, needless to say any pointers from the dev team on the right
> direction to look within the codebase would be much appreciated! :)
>
> Thanks in advance for everyone’s/anyone's time and help - am not an
> Airflow expert, but am hopefully learning quickly enough to help resolve
> this issue (if I am ‘barking up the right tree’ with this bug number…)
>
> Cheers,
> J
>
>