You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Richard Lee (JIRA)" <ji...@apache.org> on 2017/05/04 07:17:04 UTC

[jira] [Created] (AIRFLOW-1171) Encoding error for non latin-1 Postgres database

Richard Lee created AIRFLOW-1171:
------------------------------------

             Summary: Encoding error for non latin-1 Postgres database
                 Key: AIRFLOW-1171
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1171
             Project: Apache Airflow
          Issue Type: Bug
          Components: db, hooks
    Affects Versions: Airflow 1.8
         Environment: macOS 10.12.5
Python 2.7.12
Postgres 9.6.1

However, these are irrelevant to this issue.
            Reporter: Richard Lee
            Assignee: Richard Lee


There's [a known issue|https://github.com/psycopg/psycopg2/issues/331] from psycopg2 that Airflow ignores the encoding settings from db by default and which results in encoding error if there's any non latin-1 content in database cell.

Reference stack trace:
{code}
  File "dags/recipe_hourly_pageviews.py", line 73, in <module>
    dag.cli()
  File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/models.py", line 3339, in cli
    args.func(args, self)
  File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/bin/cli.py", line 585, in test
    ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
  File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
    result = func(*args, **kwargs)
  File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
    result = task_copy.execute(context=context)
  File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/operators/generic_transfer.py", l
ine 78, in execute
    destination_hook.insert_rows(table=self.destination_table, rows=results)
  File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 215, i
n insert_rows
    l.append(self._serialize_cell(cell, conn))
  File "/Users/dlackty/.pyenv/versions/2.7.12/lib/python2.7/site-packages/airflow/hooks/postgres_hook.py", line 70,
 in _serialize_cell
    return psycopg2.extensions.adapt(cell).getquoted().decode('utf-8')
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 6-10: ordinal not in range(256)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)