You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Joseph (Jira)" <ji...@apache.org> on 2019/10/15 23:37:00 UTC
[jira] [Created] (AIRFLOW-5664) postgres_to_gcs operator drops
milliseconds from timestamps
Joseph created AIRFLOW-5664:
-------------------------------
Summary: postgres_to_gcs operator drops milliseconds from timestamps
Key: AIRFLOW-5664
URL: https://issues.apache.org/jira/browse/AIRFLOW-5664
Project: Apache Airflow
Issue Type: Bug
Components: operators
Affects Versions: 1.10.5
Reporter: Joseph
Postgres stores timestamps with microsecond resolution. When using the postgres_to_gcs operator, timestamps are converted to epoch/unix time using the datetime.timetuple() method. This method drops the microseconds and so you'll end up with a storage object that looks like this:
{code:java}
{"id": 1, "last_modified": 1571038537.0}
{"id": 2, "last_modified": 1571038537.0}
{"id": 3, "last_modified": 1571038537.0}
{code}
When it should look like this:
{code:java}
{"id": 1, "last_modified": 1571038537.123}
{"id": 2, "last_modified": 1571038537.400}
{"id": 3, "last_modified": 1571038537.455}
{code}
It would be useful to keep the timestamps' full resolution.
I believe the same issue may occur with airflow.operators.mysql_to_gcs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)