You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2016/08/21 18:33:21 UTC
[jira] [Commented] (AIRFLOW-447) Python 3 map object cannot be
json-serialized - need lists instead
[ https://issues.apache.org/jira/browse/AIRFLOW-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429848#comment-15429848 ]
ASF subversion and git services commented on AIRFLOW-447:
---------------------------------------------------------
Commit fe037d6d627934568b7f9a3022dcbe70455799e1 in incubator-airflow's branch refs/heads/master from [~waltherg]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fe037d6 ]
[AIRFLOW-447] Store source URIs in Python 3 compatible list
In Python 2 map would generate a list, in Python 3
this is no longer true.
Downstream a job with source_uris set to a Python
3 map object cannot be json-serialized
when attempting to store the job in the database.
This fix keeps the source_uris in a json-
serializable list in both Python 2 and 3.
Closes #1754 from
waltherg/fix/python3_map_incompatibility
> Python 3 map object cannot be json-serialized - need lists instead
> ------------------------------------------------------------------
>
> Key: AIRFLOW-447
> URL: https://issues.apache.org/jira/browse/AIRFLOW-447
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
> Affects Versions: Airflow 1.7.1.3
> Environment: Python 3
> Reporter: Georg Walther
> Priority: Blocker
> Labels: easyfix
>
> On this line, `source_uris` are generated with `map`:
> https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/gcs_to_bq.py#L124
> In Python 2 `map` would return a list, however in Python 3 `map` returns a `map` object / generator.
> The Python 3 `map` object cannot be json-serialized downstream when storing corresponding job objects in the database:
> https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/bigquery_hook.py#L441
> Constructing `source_uris` with a list comprehension instead of `map` resolves the issue for Python 3 and retains compatibility with Python 2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)