[jira] [Commented] (AIRFLOW-3978) Add missing types in MySqlToGoogleCloudStorageOperator

["Tomasz Urbaszek (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16902979#comment-16902979 ] 

Tomasz Urbaszek commented on AIRFLOW-3978:
------------------------------------------

According to this comment [https://github.com/apache/airflow/pull/4802#issuecomment-468859708] :

"MySQL FIELD_TYPE (types used in the protocol) is not directly mapped to column type. For example, VARBINARY is returned as FIELD_TYPE.VAR_STRING."

 

In the version of MySQLdb used in Airflow one can see that indeed there is no conversion from BINARY / VARBINARY but STRING / VAR_STRING could be str or binary.

conversions = {
 int: Thing2Str,
 long: Thing2Str,
 float: Float2Str,
 NoneType: None2NULL,
 ArrayType: array2Str,
 bool: Bool2Str,
 Date: Thing2Literal,
 DateTimeType: DateTime2literal,
 DateTimeDeltaType: DateTimeDelta2literal,
 str: Thing2Literal, # default
 set: Set2Str,

 FIELD_TYPE.TINY: int,
 FIELD_TYPE.SHORT: int,
 FIELD_TYPE.LONG: long,
 FIELD_TYPE.FLOAT: float,
 FIELD_TYPE.DOUBLE: float,
 FIELD_TYPE.DECIMAL: float,
 FIELD_TYPE.NEWDECIMAL: float,
 FIELD_TYPE.LONGLONG: long,
 FIELD_TYPE.INT24: int,
 FIELD_TYPE.YEAR: int,
 FIELD_TYPE.SET: Str2Set,
 FIELD_TYPE.TIMESTAMP: mysql_timestamp_converter,
 FIELD_TYPE.DATETIME: DateTime_or_None,
 FIELD_TYPE.TIME: TimeDelta_or_None,
 FIELD_TYPE.DATE: Date_or_None,

 *FIELD_TYPE.TINY_BLOB: _bytes_or_str,*
 *FIELD_TYPE.MEDIUM_BLOB: _bytes_or_str,*
 *FIELD_TYPE.LONG_BLOB: _bytes_or_str,*
 *FIELD_TYPE.BLOB: _bytes_or_str,*
 *FIELD_TYPE.STRING: _bytes_or_str,*
 *FIELD_TYPE.VAR_STRING: _bytes_or_str,*
 *FIELD_TYPE.VARCHAR: _bytes_or_str,*
}

 

 

 

> Add missing types in MySqlToGoogleCloudStorageOperator
> ------------------------------------------------------
>
>                 Key: AIRFLOW-3978
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3978
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: gcp
>    Affects Versions: 1.10.2
>            Reporter: Roster
>            Assignee: Roster
>            Priority: Minor
>              Labels: gcs
>
> There fields are missing and can not be mapped: 
> TIME, BINARY , VARBINARY



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)