[jira] [Created] (AIRFLOW-2939) `set` fails in case of `exisiting_files is None` and in case of `json.dumps`

["Kiyoshi Nomo (JIRA)" <ji...@apache.org>]

Kiyoshi Nomo created AIRFLOW-2939:
-------------------------------------

             Summary: `set` fails in case of `exisiting_files is None` and in case of `json.dumps`
                 Key: AIRFLOW-2939
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2939
             Project: Apache Airflow
          Issue Type: Bug
          Components: operators
    Affects Versions: 1.10
            Reporter: Kiyoshi Nomo


h1. Problems
h2. TypeError: 'NoneType' object is not iterable

[https://github.com/apache/incubator-airflow/blob/06b62c42b0b55ea55b86b130317594738d2f36a2/airflow/contrib/operators/gcs_to_s3.py#L91]

 
{code:java}
>>> set(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not iterable
{code}
 
h2. TypeError: set(['a']) is not JSON serializable

[https://github.com/apache/incubator-airflow/blob/b78c7fb8512f7a40f58b46530e9b3d5562fe84ea/airflow/models.py#L4483]

 
{code:python}
>>> json.dumps(set(['a']))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/__init__.py", line 244, in dumps
return _default_encoder.encode(obj)
File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
File "/usr/local/opt/pyenv/versions/2.7.11/lib/python2.7/json/encoder.py", line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: set(['a']) is not JSON serializable
{code}
 
h1. Solution
 * Check that the existing fils is not None.
 * Convert it to the `set` and return it to the `list` after get to the difference of files.

{code:python}
if existing_files is not None:
    files = list(set(files) - set(existing_files))
{code}
 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)