[jira] [Reopened] (AIRFLOW-1171) Encoding error for non latin-1 Postgres database

["Aizhamal Nurmamat kyzy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aizhamal Nurmamat kyzy reopened AIRFLOW-1171:
---------------------------------------------

Reopening to do component refactor. Will resolve after moving.

> Encoding error for non latin-1 Postgres database
> ------------------------------------------------
>
>                 Key: AIRFLOW-1171
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1171
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: db, hooks
>    Affects Versions: 1.8.0
>         Environment: macOS 10.12.5
> Python 2.7.12
> Postgres 9.6.1
> However, these are irrelevant to this issue.
>            Reporter: Richard Lee
>            Assignee: Richard Lee
>            Priority: Major
>
> There's [a known issue|https://github.com/psycopg/psycopg2/issues/331] from psycopg2 that Airflow ignores the encoding settings from db by default and which results in encoding error if there's any non latin-1 content in database cell.
> Reference stack trace:
> {code}
>   File "dags/recipe_hourly_pageviews.py", line 73, in <module>
>     dag.cli()
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/models.py", line 3339, in cli
>     args.func(args, self)
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/bin/cli.py", line 585, in test
>     ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
>     result = func(*args, **kwargs)
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
>     result = task_copy.execute(context=context)
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/operators/generic_transfer.py", l
> ine 78, in execute
>     destination_hook.insert_rows(table=self.destination_table, rows=results)
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 215, i
> n insert_rows
>     l.append(self._serialize_cell(cell, conn))
>   File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/hooks/postgres_hook.py", line 70,
>  in _serialize_cell
>     return psycopg2.extensions.adapt(cell).getquoted().decode('utf-8')
> UnicodeEncodeError: 'latin-1' codec can't encode characters in position 6-10: ordinal not in range(256)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)