You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Joy Gao (JIRA)" <ji...@apache.org> on 2017/09/14 20:17:00 UTC
[jira] [Updated] (AIRFLOW-1613) Make
MySqlToGoogleCloudStorageOperator compaitible with python3
[ https://issues.apache.org/jira/browse/AIRFLOW-1613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joy Gao updated AIRFLOW-1613:
-----------------------------
Description:
In Python 3, map(...) returns an iterator, which can only be iterated over once.
Therefore the current implementation will return an empty list after the first iteration of schema:
{code:python}
schema = map(lambda schema_tuple: schema_tuple[0], cursor.description)
file_no = 0
tmp_file_handle = NamedTemporaryFile(delete=True)
tmp_file_handles = {self.filename.format(file_no): tmp_file_handle}
for row in cursor:
# Convert datetime objects to utc seconds, and decimals to floats
row = map(self.convert_types, row)
row_dict = dict(zip(schema, row))
{code}
Moving it inside the loop for re-use.
was:
In Python 3, map(...) returns an iterator, which can only be iterated over once.
Therefore the current implementation will return an empty list after the first iteration of schema:
{code:python}
schema = map(lambda schema_tuple: schema_tuple[0], cursor.description)
file_no = 0
tmp_file_handle = NamedTemporaryFile(delete=True)
tmp_file_handles = {self.filename.format(file_no): tmp_file_handle}
for row in cursor:
# Convert datetime objects to utc seconds, and decimals to floats
row = map(self.convert_types, row)
row_dict = dict(zip(schema, row))
{code}
Moving it inside the loop for re-use.
> Make MySqlToGoogleCloudStorageOperator compaitible with python3
> ---------------------------------------------------------------
>
> Key: AIRFLOW-1613
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1613
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
> Reporter: Joy Gao
>
> In Python 3, map(...) returns an iterator, which can only be iterated over once.
> Therefore the current implementation will return an empty list after the first iteration of schema:
> {code:python}
> schema = map(lambda schema_tuple: schema_tuple[0], cursor.description)
> file_no = 0
> tmp_file_handle = NamedTemporaryFile(delete=True)
> tmp_file_handles = {self.filename.format(file_no): tmp_file_handle}
> for row in cursor:
> # Convert datetime objects to utc seconds, and decimals to floats
> row = map(self.convert_types, row)
> row_dict = dict(zip(schema, row))
> {code}
> Moving it inside the loop for re-use.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)