[jira] [Commented] (AIRFLOW-1171) Encoding error for non latin-1 Postgres database

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707136#comment-16707136 ] 

ASF GitHub Bot commented on AIRFLOW-1171:
-----------------------------------------

kaxil closed pull request #2273: [AIRFLOW-1171] Fix up encoding for Postgres
URL: https://github.com/apache/incubator-airflow/pull/2273
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/hooks/postgres_hook.py b/airflow/hooks/postgres_hook.py
index 4b460c1158..6ab9ea7110 100644
--- a/airflow/hooks/postgres_hook.py
+++ b/airflow/hooks/postgres_hook.py
@@ -67,4 +67,9 @@ def _serialize_cell(cell, conn):
         :rtype: str
         """
 
-        return psycopg2.extensions.adapt(cell).getquoted().decode('utf-8')
+        adapted = psycopg2.extensions.adapt(cell)
+        try:
+            adapted.prepare(conn)
+        except AttributeError:
+            pass
+        return adapted.getquoted()


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Encoding error for non latin-1 Postgres database
> ------------------------------------------------
>
>                 Key: AIRFLOW-1171
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1171
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: db, hooks
>    Affects Versions: 1.8.0
>         Environment: macOS 10.12.5
> Python 2.7.12
> Postgres 9.6.1
> However, these are irrelevant to this issue.
>            Reporter: Richard Lee
>            Assignee: Richard Lee
>            Priority: Major
>
> There's [a known issue|https://github.com/psycopg/psycopg2/issues/331] from psycopg2 that Airflow ignores the encoding settings from db by default and which results in encoding error if there's any non latin-1 content in database cell.
> Reference stack trace:
> {code}
>   File "dags/recipe_hourly_pageviews.py", line 73, in <module>
>     dag.cli()
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/models.py", line 3339, in cli
>     args.func(args, self)
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/bin/cli.py", line 585, in test
>     ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
>     result = func(*args, **kwargs)
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
>     result = task_copy.execute(context=context)
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/operators/generic_transfer.py", l
> ine 78, in execute
>     destination_hook.insert_rows(table=self.destination_table, rows=results)
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 215, i
> n insert_rows
>     l.append(self._serialize_cell(cell, conn))
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/hooks/postgres_hook.py", line 70,
>  in _serialize_cell
>     return psycopg2.extensions.adapt(cell).getquoted().decode('utf-8')
> UnicodeEncodeError: 'latin-1' codec can't encode characters in position 6-10: ordinal not in range(256)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)