You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Tomasz Urbaszek (JIRA)" <ji...@apache.org> on 2019/08/08 13:30:00 UTC
[jira] [Comment Edited] (AIRFLOW-3978) Add missing types in
MySqlToGoogleCloudStorageOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902979#comment-16902979 ]
Tomasz Urbaszek edited comment on AIRFLOW-3978 at 8/8/19 1:29 PM:
------------------------------------------------------------------
According to this comment [https://github.com/apache/airflow/pull/4802#issuecomment-468859708] :
"MySQL FIELD_TYPE (types used in the protocol) is not directly mapped to column type. For example, VARBINARY is returned as FIELD_TYPE.VAR_STRING."
In the version of MySQLdb used in Airflow one can see that indeed there is no conversion from BINARY / VARBINARY but STRING / VAR_STRING could be str or binary.
{code:python}
conversions = {
int: Thing2Str,
long: Thing2Str,
float: Float2Str,
NoneType: None2NULL,
ArrayType: array2Str,
bool: Bool2Str,
Date: Thing2Literal,
DateTimeType: DateTime2literal,
DateTimeDeltaType: DateTimeDelta2literal,
str: Thing2Literal, # default
set: Set2Str,
FIELD_TYPE.TINY: int,
FIELD_TYPE.SHORT: int,
FIELD_TYPE.LONG: long,
FIELD_TYPE.FLOAT: float,
FIELD_TYPE.DOUBLE: float,
FIELD_TYPE.DECIMAL: float,
FIELD_TYPE.NEWDECIMAL: float,
FIELD_TYPE.LONGLONG: long,
FIELD_TYPE.INT24: int,
FIELD_TYPE.YEAR: int,
FIELD_TYPE.SET: Str2Set,
FIELD_TYPE.TIMESTAMP: mysql_timestamp_converter,
FIELD_TYPE.DATETIME: DateTime_or_None,
FIELD_TYPE.TIME: TimeDelta_or_None,
FIELD_TYPE.DATE: Date_or_None,
*FIELD_TYPE.TINY_BLOB: _bytes_or_str,*
*FIELD_TYPE.MEDIUM_BLOB: _bytes_or_str,*
*FIELD_TYPE.LONG_BLOB: _bytes_or_str,*
*FIELD_TYPE.BLOB: _bytes_or_str,*
*FIELD_TYPE.STRING: _bytes_or_str,*
*FIELD_TYPE.VAR_STRING: _bytes_or_str,*
*FIELD_TYPE.VARCHAR: _bytes_or_str,*
}
{code}
was (Author: urbaszek):
According to this comment [https://github.com/apache/airflow/pull/4802#issuecomment-468859708] :
"MySQL FIELD_TYPE (types used in the protocol) is not directly mapped to column type. For example, VARBINARY is returned as FIELD_TYPE.VAR_STRING."
In the version of MySQLdb used in Airflow one can see that indeed there is no conversion from BINARY / VARBINARY but STRING / VAR_STRING could be str or binary.
conversions = {
int: Thing2Str,
long: Thing2Str,
float: Float2Str,
NoneType: None2NULL,
ArrayType: array2Str,
bool: Bool2Str,
Date: Thing2Literal,
DateTimeType: DateTime2literal,
DateTimeDeltaType: DateTimeDelta2literal,
str: Thing2Literal, # default
set: Set2Str,
FIELD_TYPE.TINY: int,
FIELD_TYPE.SHORT: int,
FIELD_TYPE.LONG: long,
FIELD_TYPE.FLOAT: float,
FIELD_TYPE.DOUBLE: float,
FIELD_TYPE.DECIMAL: float,
FIELD_TYPE.NEWDECIMAL: float,
FIELD_TYPE.LONGLONG: long,
FIELD_TYPE.INT24: int,
FIELD_TYPE.YEAR: int,
FIELD_TYPE.SET: Str2Set,
FIELD_TYPE.TIMESTAMP: mysql_timestamp_converter,
FIELD_TYPE.DATETIME: DateTime_or_None,
FIELD_TYPE.TIME: TimeDelta_or_None,
FIELD_TYPE.DATE: Date_or_None,
*FIELD_TYPE.TINY_BLOB: _bytes_or_str,*
*FIELD_TYPE.MEDIUM_BLOB: _bytes_or_str,*
*FIELD_TYPE.LONG_BLOB: _bytes_or_str,*
*FIELD_TYPE.BLOB: _bytes_or_str,*
*FIELD_TYPE.STRING: _bytes_or_str,*
*FIELD_TYPE.VAR_STRING: _bytes_or_str,*
*FIELD_TYPE.VARCHAR: _bytes_or_str,*
}
> Add missing types in MySqlToGoogleCloudStorageOperator
> ------------------------------------------------------
>
> Key: AIRFLOW-3978
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3978
> Project: Apache Airflow
> Issue Type: Bug
> Components: gcp
> Affects Versions: 1.10.2
> Reporter: Roster
> Assignee: Roster
> Priority: Minor
> Labels: gcs
>
> There fields are missing and can not be mapped:
> TIME, BINARY , VARBINARY
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)