You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Sumit Maheshwari <su...@gmail.com> on 2017/12/06 14:13:00 UTC
Xcom_push is broken - probably after the timezone awareness
Folks,
I've some testes, which used to pass earlier, but recently they started
failing. When I looked closely it appeared that xcom_push is behaving
differently.
So this is is the current code, which throws the error mentioned below to it
DEFAULT_DATE = datetime(2017, 1, 1)
ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
ti.xcom_push('abcd', 12345)
File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line
31, in process_bind_param
raise ValueError('naive datetime is disallowed')
StatementError: (exceptions.ValueError) naive datetime is disallowed
[SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date
= ? AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
When I changed the execution date param to a string, it fails with other
error.
ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
ti.xcom_push('abcd', 12345)
File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line
29, in process_bind_param
repr(value))
StatementError: (exceptions.TypeError) expected datetime.datetime, not
'2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ?
AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?']
[parameters: [{}]]
Somehow I feel that it is linked with *sqlalchemy_utc* library, though
I may be wrong and its just missing some timezone related settings.
Please let me know if anyone else has faced this issue as well.
Thanks,
Sumit
Re: Xcom_push is broken - probably after the timezone awareness
Posted by Sumit Maheshwari <su...@gmail.com>.
Sounds perfect. Will fix my tests as of now and try to create a patch as
well.
On Wed, Dec 6, 2017 at 7:58 PM, Bolke de Bruin <bd...@gmail.com> wrote:
> You are directly instantiating a TaskInstance which does not verify
> against naive dates being passed, not a DAG. DAGs with naive dates will
> work fine.
>
> So for TaskInstances, instantiated outside Airflow itself (ie your python
> file) it will fail on *master* given no patch. So, I guess a patch is
> very welcome if you are relying on this behaviour.
>
> Cheers
> Bolke
>
> On 6 Dec 2017, at 15:25, Sumit Maheshwari <su...@gmail.com> wrote:
>
> Thanks Bolke for the information.
>
> BTW will existing dags which are using native datetime start failing after
> this change, given no patch is done??
>
> Just to be precise, dags which may be using methods like xcom push/pull?
>
> On Dec 6, 2017 7:49 PM, "Bolke de Bruin" <bd...@gmail.com> wrote:
>
>> You should not use a naive datetime as it mentioned. So instead of of
>> using “from datetime import datetime” use “from airflow.utils.timezone
>> import datetime”. This sets the timezone information that is required to
>> the default configured in airflow.cfg.
>>
>> A patch that sets the default timezone for TaskInstance is also helpful
>> in this case and will be welcomed. How to do it you can find in the DAG
>> class.
>>
>> Bolke.
>>
>> On 6 Dec 2017, at 15:13, Sumit Maheshwari <su...@gmail.com> wrote:
>>
>> Folks,
>>
>> I've some testes, which used to pass earlier, but recently they started
>> failing. When I looked closely it appeared that xcom_push is behaving
>> differently.
>>
>>
>> So this is is the current code, which throws the error mentioned below to
>> it
>>
>> DEFAULT_DATE = datetime(2017, 1, 1)
>>
>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
>> ti.xcom_push('abcd', 12345)
>>
>>
>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31, in process_bind_param
>>
>> raise ValueError('naive datetime is disallowed')
>> StatementError: (exceptions.ValueError) naive datetime is disallowed [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
>>
>>
>> When I changed the execution date param to a string, it fails with other
>> error.
>>
>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
>> ti.xcom_push('abcd', 12345)
>>
>>
>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29, in process_bind_param
>> repr(value))
>> StatementError: (exceptions.TypeError) expected datetime.datetime, not '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
>>
>>
>> Somehow I feel that it is linked with *sqlalchemy_utc* library, though I may be wrong and its just missing some timezone related settings.
>>
>> Please let me know if anyone else has faced this issue as well.
>>
>>
>> Thanks,
>>
>> Sumit
>>
>>
>>
>>
>>
>
Re: Xcom_push is broken - probably after the timezone awareness
Posted by Bolke de Bruin <bd...@gmail.com>.
You are directly instantiating a TaskInstance which does not verify against naive dates being passed, not a DAG. DAGs with naive dates will work fine.
So for TaskInstances, instantiated outside Airflow itself (ie your python file) it will fail on master given no patch. So, I guess a patch is very welcome if you are relying on this behaviour.
Cheers
Bolke
> On 6 Dec 2017, at 15:25, Sumit Maheshwari <su...@gmail.com> wrote:
>
> Thanks Bolke for the information.
>
> BTW will existing dags which are using native datetime start failing after this change, given no patch is done??
>
> Just to be precise, dags which may be using methods like xcom push/pull?
>
> On Dec 6, 2017 7:49 PM, "Bolke de Bruin" <bdbruin@gmail.com <ma...@gmail.com>> wrote:
> You should not use a naive datetime as it mentioned. So instead of of using “from datetime import datetime” use “from airflow.utils.timezone import datetime”. This sets the timezone information that is required to the default configured in airflow.cfg.
>
> A patch that sets the default timezone for TaskInstance is also helpful in this case and will be welcomed. How to do it you can find in the DAG class.
>
> Bolke.
>
>> On 6 Dec 2017, at 15:13, Sumit Maheshwari <sumeet.manit@gmail.com <ma...@gmail.com>> wrote:
>>
>> Folks,
>>
>> I've some testes, which used to pass earlier, but recently they started failing. When I looked closely it appeared that xcom_push is behaving differently.
>>
>>
>> So this is is the current code, which throws the error mentioned below to it
>> DEFAULT_DATE = datetime(2017, 1, 1)
>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
>> ti.xcom_push('abcd', 12345)
>>
>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31, in process_bind_param
>> raise ValueError('naive datetime is disallowed')
>> StatementError: (exceptions.ValueError) naive datetime is disallowed [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
>>
>> When I changed the execution date param to a string, it fails with other error.
>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
>> ti.xcom_push('abcd', 12345)
>>
>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29, in process_bind_param
>> repr(value))
>> StatementError: (exceptions.TypeError) expected datetime.datetime, not '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
>>
>> Somehow I feel that it is linked with sqlalchemy_utc library, though I may be wrong and its just missing some timezone related settings.
>> Please let me know if anyone else has faced this issue as well.
>>
>> Thanks,
>> Sumit
>>
>>
>
Re: Xcom_push is broken - probably after the timezone awareness
Posted by Sumit Maheshwari <su...@gmail.com>.
Thanks Bolke for the information.
BTW will existing dags which are using native datetime start failing after
this change, given no patch is done??
Just to be precise, dags which may be using methods like xcom push/pull?
On Dec 6, 2017 7:49 PM, "Bolke de Bruin" <bd...@gmail.com> wrote:
> You should not use a naive datetime as it mentioned. So instead of of
> using “from datetime import datetime” use “from airflow.utils.timezone
> import datetime”. This sets the timezone information that is required to
> the default configured in airflow.cfg.
>
> A patch that sets the default timezone for TaskInstance is also helpful in
> this case and will be welcomed. How to do it you can find in the DAG class.
>
> Bolke.
>
> On 6 Dec 2017, at 15:13, Sumit Maheshwari <su...@gmail.com> wrote:
>
> Folks,
>
> I've some testes, which used to pass earlier, but recently they started
> failing. When I looked closely it appeared that xcom_push is behaving
> differently.
>
>
> So this is is the current code, which throws the error mentioned below to
> it
>
> DEFAULT_DATE = datetime(2017, 1, 1)
>
> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
> ti.xcom_push('abcd', 12345)
>
>
> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31, in process_bind_param
>
> raise ValueError('naive datetime is disallowed')
> StatementError: (exceptions.ValueError) naive datetime is disallowed [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
>
>
> When I changed the execution date param to a string, it fails with other
> error.
>
> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
> ti.xcom_push('abcd', 12345)
>
>
> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29, in process_bind_param
> repr(value))
> StatementError: (exceptions.TypeError) expected datetime.datetime, not '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
>
>
> Somehow I feel that it is linked with *sqlalchemy_utc* library, though I may be wrong and its just missing some timezone related settings.
>
> Please let me know if anyone else has faced this issue as well.
>
>
> Thanks,
>
> Sumit
>
>
>
>
>
Re: Xcom_push is broken - probably after the timezone awareness
Posted by Boris Tyukin <bo...@boristyukin.com>.
:) you rock as always Bolke!!
On Wed, Dec 6, 2017 at 11:17 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> 1. Naive dates are supported for initalization of DAGs and will be, after
> the patch of Sumit, for directly instantiated TaskInstances. XCom can store
> arbitrary information so it is not affected.
> 2. Any arithmetic you do on dates will need to happen with a timezone
> associated with it. This rare, you normally do date + datetime. However if
> you want to compare dates you will need to make it timezone aware.
> 3. Timezone support will not be in 1.9.0
> 4. Logging.info will be supported in 1.9.0 final
>
>
> So no reason to panic I would say ;-)
>
> Bolke
>
> Verstuurd vanaf mijn iPad
>
> > Op 6 dec. 2017 om 16:35 heeft Boris Tyukin <bo...@boristyukin.com> het
> volgende geschreven:
> >
> > hm...can you elaborate why we cannot use "from datetime import
> datetime".
> > it worked before just fine :) i was under an impression that any pickle
> > serializable type can be using with xcom.
> >
> > Along with a recent change in logs (logging.info does not write to log
> > files anymore), this makes upgrade to 1.9 quite difficult.
> >
> >> On Wed, Dec 6, 2017 at 9:19 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> You should not use a naive datetime as it mentioned. So instead of of
> >> using “from datetime import datetime” use “from airflow.utils.timezone
> >> import datetime”. This sets the timezone information that is required to
> >> the default configured in airflow.cfg.
> >>
> >> A patch that sets the default timezone for TaskInstance is also helpful
> in
> >> this case and will be welcomed. How to do it you can find in the DAG
> class.
> >>
> >> Bolke.
> >>
> >>> On 6 Dec 2017, at 15:13, Sumit Maheshwari <su...@gmail.com>
> >> wrote:
> >>>
> >>> Folks,
> >>>
> >>> I've some testes, which used to pass earlier, but recently they started
> >> failing. When I looked closely it appeared that xcom_push is behaving
> >> differently.
> >>>
> >>>
> >>> So this is is the current code, which throws the error mentioned below
> >> to it
> >>> DEFAULT_DATE = datetime(2017, 1, 1)
> >>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
> >>> ti.xcom_push('abcd', 12345)
> >>>
> >>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31,
> >> in process_bind_param
> >>> raise ValueError('naive datetime is disallowed')
> >>> StatementError: (exceptions.ValueError) naive datetime is disallowed
> >> [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date =
> ?
> >> AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
> >>>
> >>> When I changed the execution date param to a string, it fails with
> other
> >> error.
> >>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
> >>> ti.xcom_push('abcd', 12345)
> >>>
> >>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29,
> >> in process_bind_param
> >>> repr(value))
> >>> StatementError: (exceptions.TypeError) expected datetime.datetime, not
> >> '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND
> >> xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?']
> >> [parameters: [{}]]
> >>>
> >>> Somehow I feel that it is linked with sqlalchemy_utc library, though I
> >> may be wrong and its just missing some timezone related settings.
> >>> Please let me know if anyone else has faced this issue as well.
> >>>
> >>> Thanks,
> >>> Sumit
> >>>
> >>>
> >>
> >>
>
Re: Xcom_push is broken - probably after the timezone awareness
Posted by Bolke de Bruin <bd...@gmail.com>.
1. Naive dates are supported for initalization of DAGs and will be, after the patch of Sumit, for directly instantiated TaskInstances. XCom can store arbitrary information so it is not affected.
2. Any arithmetic you do on dates will need to happen with a timezone associated with it. This rare, you normally do date + datetime. However if you want to compare dates you will need to make it timezone aware.
3. Timezone support will not be in 1.9.0
4. Logging.info will be supported in 1.9.0 final
So no reason to panic I would say ;-)
Bolke
Verstuurd vanaf mijn iPad
> Op 6 dec. 2017 om 16:35 heeft Boris Tyukin <bo...@boristyukin.com> het volgende geschreven:
>
> hm...can you elaborate why we cannot use "from datetime import datetime".
> it worked before just fine :) i was under an impression that any pickle
> serializable type can be using with xcom.
>
> Along with a recent change in logs (logging.info does not write to log
> files anymore), this makes upgrade to 1.9 quite difficult.
>
>> On Wed, Dec 6, 2017 at 9:19 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> You should not use a naive datetime as it mentioned. So instead of of
>> using “from datetime import datetime” use “from airflow.utils.timezone
>> import datetime”. This sets the timezone information that is required to
>> the default configured in airflow.cfg.
>>
>> A patch that sets the default timezone for TaskInstance is also helpful in
>> this case and will be welcomed. How to do it you can find in the DAG class.
>>
>> Bolke.
>>
>>> On 6 Dec 2017, at 15:13, Sumit Maheshwari <su...@gmail.com>
>> wrote:
>>>
>>> Folks,
>>>
>>> I've some testes, which used to pass earlier, but recently they started
>> failing. When I looked closely it appeared that xcom_push is behaving
>> differently.
>>>
>>>
>>> So this is is the current code, which throws the error mentioned below
>> to it
>>> DEFAULT_DATE = datetime(2017, 1, 1)
>>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
>>> ti.xcom_push('abcd', 12345)
>>>
>>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31,
>> in process_bind_param
>>> raise ValueError('naive datetime is disallowed')
>>> StatementError: (exceptions.ValueError) naive datetime is disallowed
>> [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ?
>> AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
>>>
>>> When I changed the execution date param to a string, it fails with other
>> error.
>>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
>>> ti.xcom_push('abcd', 12345)
>>>
>>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29,
>> in process_bind_param
>>> repr(value))
>>> StatementError: (exceptions.TypeError) expected datetime.datetime, not
>> '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND
>> xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?']
>> [parameters: [{}]]
>>>
>>> Somehow I feel that it is linked with sqlalchemy_utc library, though I
>> may be wrong and its just missing some timezone related settings.
>>> Please let me know if anyone else has faced this issue as well.
>>>
>>> Thanks,
>>> Sumit
>>>
>>>
>>
>>
Re: Xcom_push is broken - probably after the timezone awareness
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
Serialising XCom via pickle is being deprecated for Airflow 2 btw:
https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;a=commitdiff;h=4cf904c;hp=984a87c0cb685ea4dfa765cc4f4a23c9058b3965 <https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;a=commitdiff;h=4cf904c;hp=984a87c0cb685ea4dfa765cc4f4a23c9058b3965>
> +- The pickle type for XCom messages has been replaced by json to prevent RCE attacks.
> + Note that JSON serialization is stricter than pickling, so if you want to e.g. pass
> + raw bytes through XCom you must encode them using an encoding like base64.
> + By default pickling is still enabled until Airflow 2.0. To disable it
> + Set enable_xcom_pickling = False in your Airflow config.
Was your test performed against the release candidate or master? I thought the timezone-aware date change was only in the later.
-ash
> On 6 Dec 2017, at 15:35, Boris Tyukin <bo...@boristyukin.com> wrote:
>
> hm...can you elaborate why we cannot use "from datetime import datetime".
> it worked before just fine :) i was under an impression that any pickle
> serializable type can be using with xcom.
>
> Along with a recent change in logs (logging.info does not write to log
> files anymore), this makes upgrade to 1.9 quite difficult.
>
> On Wed, Dec 6, 2017 at 9:19 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>
>> You should not use a naive datetime as it mentioned. So instead of of
>> using “from datetime import datetime” use “from airflow.utils.timezone
>> import datetime”. This sets the timezone information that is required to
>> the default configured in airflow.cfg.
>>
>> A patch that sets the default timezone for TaskInstance is also helpful in
>> this case and will be welcomed. How to do it you can find in the DAG class.
>>
>> Bolke.
>>
>>> On 6 Dec 2017, at 15:13, Sumit Maheshwari <su...@gmail.com>
>> wrote:
>>>
>>> Folks,
>>>
>>> I've some testes, which used to pass earlier, but recently they started
>> failing. When I looked closely it appeared that xcom_push is behaving
>> differently.
>>>
>>>
>>> So this is is the current code, which throws the error mentioned below
>> to it
>>> DEFAULT_DATE = datetime(2017, 1, 1)
>>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
>>> ti.xcom_push('abcd', 12345)
>>>
>>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31,
>> in process_bind_param
>>> raise ValueError('naive datetime is disallowed')
>>> StatementError: (exceptions.ValueError) naive datetime is disallowed
>> [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ?
>> AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
>>>
>>> When I changed the execution date param to a string, it fails with other
>> error.
>>> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
>>> ti.xcom_push('abcd', 12345)
>>>
>>> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29,
>> in process_bind_param
>>> repr(value))
>>> StatementError: (exceptions.TypeError) expected datetime.datetime, not
>> '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND
>> xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?']
>> [parameters: [{}]]
>>>
>>> Somehow I feel that it is linked with sqlalchemy_utc library, though I
>> may be wrong and its just missing some timezone related settings.
>>> Please let me know if anyone else has faced this issue as well.
>>>
>>> Thanks,
>>> Sumit
>>>
>>>
>>
>>
Re: Xcom_push is broken - probably after the timezone awareness
Posted by Boris Tyukin <bo...@boristyukin.com>.
hm...can you elaborate why we cannot use "from datetime import datetime".
it worked before just fine :) i was under an impression that any pickle
serializable type can be using with xcom.
Along with a recent change in logs (logging.info does not write to log
files anymore), this makes upgrade to 1.9 quite difficult.
On Wed, Dec 6, 2017 at 9:19 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> You should not use a naive datetime as it mentioned. So instead of of
> using “from datetime import datetime” use “from airflow.utils.timezone
> import datetime”. This sets the timezone information that is required to
> the default configured in airflow.cfg.
>
> A patch that sets the default timezone for TaskInstance is also helpful in
> this case and will be welcomed. How to do it you can find in the DAG class.
>
> Bolke.
>
> > On 6 Dec 2017, at 15:13, Sumit Maheshwari <su...@gmail.com>
> wrote:
> >
> > Folks,
> >
> > I've some testes, which used to pass earlier, but recently they started
> failing. When I looked closely it appeared that xcom_push is behaving
> differently.
> >
> >
> > So this is is the current code, which throws the error mentioned below
> to it
> > DEFAULT_DATE = datetime(2017, 1, 1)
> > ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
> > ti.xcom_push('abcd', 12345)
> >
> > File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31,
> in process_bind_param
> > raise ValueError('naive datetime is disallowed')
> > StatementError: (exceptions.ValueError) naive datetime is disallowed
> [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ?
> AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
> >
> > When I changed the execution date param to a string, it fails with other
> error.
> > ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
> > ti.xcom_push('abcd', 12345)
> >
> > File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29,
> in process_bind_param
> > repr(value))
> > StatementError: (exceptions.TypeError) expected datetime.datetime, not
> '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND
> xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?']
> [parameters: [{}]]
> >
> > Somehow I feel that it is linked with sqlalchemy_utc library, though I
> may be wrong and its just missing some timezone related settings.
> > Please let me know if anyone else has faced this issue as well.
> >
> > Thanks,
> > Sumit
> >
> >
>
>
Re: Xcom_push is broken - probably after the timezone awareness
Posted by Bolke de Bruin <bd...@gmail.com>.
You should not use a naive datetime as it mentioned. So instead of of using “from datetime import datetime” use “from airflow.utils.timezone import datetime”. This sets the timezone information that is required to the default configured in airflow.cfg.
A patch that sets the default timezone for TaskInstance is also helpful in this case and will be welcomed. How to do it you can find in the DAG class.
Bolke.
> On 6 Dec 2017, at 15:13, Sumit Maheshwari <su...@gmail.com> wrote:
>
> Folks,
>
> I've some testes, which used to pass earlier, but recently they started failing. When I looked closely it appeared that xcom_push is behaving differently.
>
>
> So this is is the current code, which throws the error mentioned below to it
> DEFAULT_DATE = datetime(2017, 1, 1)
> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE)
> ti.xcom_push('abcd', 12345)
>
> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 31, in process_bind_param
> raise ValueError('naive datetime is disallowed')
> StatementError: (exceptions.ValueError) naive datetime is disallowed [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
>
> When I changed the execution date param to a string, it fails with other error.
> ti = TaskInstance(task=task, execution_date=DEFAULT_DATE.isoformat())
> ti.xcom_push('abcd', 12345)
>
> File "build/bdist.macosx-10.11-intel/egg/sqlalchemy_utc.py", line 29, in process_bind_param
> repr(value))
> StatementError: (exceptions.TypeError) expected datetime.datetime, not '2017-01-01T00:00:00' [SQL: u'DELETE FROM xcom WHERE xcom."key" = ? AND xcom.execution_date = ? AND xcom.task_id = ? AND xcom.dag_id = ?'] [parameters: [{}]]
>
> Somehow I feel that it is linked with sqlalchemy_utc library, though I may be wrong and its just missing some timezone related settings.
> Please let me know if anyone else has faced this issue as well.
>
> Thanks,
> Sumit
>
>