You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/10 15:52:57 UTC

[GitHub] [airflow] ajbosco opened a new issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

ajbosco opened a new issue #19519:
URL: https://github.com/apache/airflow/issues/19519


   ### Apache Airflow version
   
   2.2.1 (latest released)
   
   ### Operating System
   
   Debian Buster (docker)
   
   ### Versions of Apache Airflow Providers
   
   n/a
   
   ### Deployment
   
   Other 3rd-party Helm chart
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   The DAG parses and runs fine on 2.1.3, but when we upgrade to the latest version it fails to import:
   
   ```
   Broken DAG: [/usr/local/airflow/dags/test_dag.py] Traceback (most recent call last):
     File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
       return _iterencode(o, 0)
     File "/usr/local/lib/python3.8/json/encoder.py", line 179, in default
       raise TypeError(f'Object of type {o.__class__.__name__} '
   TypeError: Object of type datetime is not JSON serializable
   ```
   
   ### What you expected to happen
   
   _No response_
   
   ### How to reproduce
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jedcunningham commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965500695


   @ajbosco, can you provide a DAG that reproduces this? The DAGs I have handy that use set params work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965538417


   ```
   @dag.task(params={"a": {1, 2, 3}, "b": datetime.now()})
   def datetime_param(intersection):
       print(intersection)
   
   datetime_param("{{ params.a }} | {{ params.b }}")
   ```
   
   could not repro with this task


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965539219


   Try pendulum datetime @dstandish 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jedcunningham commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965464493


   This should be fixed in 2.2.2 via #19267. Can you test in 2.2.2rc1 and report your results in #19515? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ajbosco commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
ajbosco commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965602956


   Welp....I think there was some issue in the database I was using. I started over from scratch and cannot reproduce this error either. My bad!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish edited a comment on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965538417


   ```
   @dag.task(params={"a": {1, 2, 3}, "b": datetime.now()})
   def datetime_param(value):
       print(value)
   
   datetime_param("{{ params.a }} | {{ params.b }}")
   ```
   
   could not repro with this task


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965604300


   @ajbosco Awesome, can you change your vote on the mailing list too, please :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965621578


   Exception being likely DAGs tha have been deleted already.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish edited a comment on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965540006


   no luck 
   ```import pendulum
   @dag.task(params={"a": {1, 2, 3}, "b": pendulum.now()})
   def datetime_param(value):
       print(value)
   
   
   datetime_param("{{ params.a }} | {{ params.b }}")
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish edited a comment on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965613892


   @ajbosco, it's not impossible that you encountered some kind of issue with upgrade, deserializing dags that were serialized _prior to_ 2.2.0.
   
   If that's what you were seeing, it would be really good if you could let us know.  I.e. we try to be backwards compatible but if there was an issue then that would be great if we could try to repro and fix.  If you think that might be what was going on and you can repro LMK.  I can try to check this too.  What version are you upgrading from?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jedcunningham closed issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
jedcunningham closed issue #19519:
URL: https://github.com/apache/airflow/issues/19519


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965621090


   Also I think we have a new "dags reserialize" coming - https://github.com/apache/airflow/pull/19471 - which is really good idea because the "serialized" form of DAGs is pretty safe to delete and recreate.
   
   I just wonder if this is not something that we should always do after/during upgrade "just in case". It might slow-down initial dag refresh, but at least we can be sure that everything in serialized dags is "correct". 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965613892


   @ajbosco, it's not impossible that you encountered some kind of issue with upgrade, deserializing dags from _prior to_ 2.2.0.
   
   If that's what you were seeing, it would be really good if you could let us know.  I.e. we try to be backwards compatible but if there was an issue then that would be great if we could try to repro and fix.  If you think that might be what was going on and you can repro LMK.  I can try to check this too.  What version are you upgrading from?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965542422


   Yup, let's just wait until then.
   
   Also please double-check with the Airflow version in the Footer of the webserver or airflow CLI to confirm that you are using 2.2.2rc1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965540790


   We need example then @ajbosco :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ajbosco closed issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
ajbosco closed issue #19519:
URL: https://github.com/apache/airflow/issues/19519


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965622487


   > I just wonder if this is not something that we should always do after/during upgrade "just in case". It might slow-down initial dag refresh, but at least we can be sure that everything in serialized dags is "correct".
   
   This I agree with completely


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish edited a comment on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
dstandish edited a comment on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965540006


   no luck 
   ```import pendulum
   @dag.task(params={"a": {1, 2, 3}, "b": pendulum.now()})
   def datetime_param(intersection):
       print(intersection)
   
   
   datetime_param("{{ params.a }} | {{ params.b }}")
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965540006


   ```import pendulum
   
   
   @dag.task(params={"a": {1, 2, 3}, "b": pendulum.now()})
   def datetime_param(intersection):
       print(intersection)
   
   
   datetime_param("{{ params.a }} | {{ params.b }}")
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jedcunningham commented on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965539872


   `pendulum.datetime` worked for me 🤷‍♂️


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] jedcunningham edited a comment on issue #19519: datetime in parameters cause "TypeError: Object of type set is not JSON serializable"

Posted by GitBox <gi...@apache.org>.
jedcunningham edited a comment on issue #19519:
URL: https://github.com/apache/airflow/issues/19519#issuecomment-965500695


   @ajbosco, can you provide a DAG that reproduces this? The DAGs I have handy that use set and datetime params work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org