You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Navya Krishnappa (JIRA)" <ji...@apache.org> on 2017/03/30 05:35:44 UTC
[jira] [Updated] (SPARK-20152) Time zone is not respected while
parsing csv for timeStampFormat "MM-dd-yyyy'T'HH:mm:ss.SSSZZ"
[ https://issues.apache.org/jira/browse/SPARK-20152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Navya Krishnappa updated SPARK-20152:
-------------------------------------
Description:
When reading the below mentioned time value by specifying the "timestampFormat": "MM-dd-yyyy'T'HH:mm:ss.SSSZZ", time zone is ignored.
Source File:
TimeColumn
03-21-2017T03:30:02Z
Source code1:
Dataset dataset = getSqlContext().read()
.option(DAWBConstant.PARSER_LIB, "commons")
.option(INFER_SCHEMA, "true")
.option(DAWBConstant.DELIMITER, ",")
.option(DAWBConstant.QUOTE, "\"")
.option(DAWBConstant.ESCAPE, "\\")
.option("timestampFormat" , "MM-dd-yyyy'T'HH:mm:ss.SSSZZ")
.option(DAWBConstant.MODE, Mode.PERMISSIVE)
.csv(sourceFile);
Result: TimeColumn [ StringType] and value is "03-21-2017T03:30:02Z", but expected result is TimeCoumn should be of "TimestampType" and should consider time zone for manipulation
Source code2:
Dataset dataset = getSqlContext().read()
.option(DAWBConstant.PARSER_LIB, "commons")
.option(INFER_SCHEMA, "true")
.option(DAWBConstant.DELIMITER, ",")
.option(DAWBConstant.QUOTE, "\"")
.option(DAWBConstant.ESCAPE, "\\")
.option("timestampFormat" , "MM-dd-yyyy'T'HH:mm:ss")
.option(DAWBConstant.MODE, Mode.PERMISSIVE)
.csv(sourceFile);
Result: TimeColumn [ TimestampType] and value is "2017-04-22 03:30:02.0", but expected result is TimeCoumn should consider time zone for manipulation
was:
When reading the below mentioned time value by specifying the "timestampFormat": "MM-dd-yyyy'T'HH:mm:ss.SSSZZ", time zone is ignored.
Sample data:
TimeColumn
03-21-2017T03:30:02Z
Result: TimeColumn [ StringType] and value is "03-21-2017T03:30:02Z"
Expected Result: TimeCoumn should be of "TimestampType"
> Time zone is not respected while parsing csv for timeStampFormat "MM-dd-yyyy'T'HH:mm:ss.SSSZZ"
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-20152
> URL: https://issues.apache.org/jira/browse/SPARK-20152
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Navya Krishnappa
>
> When reading the below mentioned time value by specifying the "timestampFormat": "MM-dd-yyyy'T'HH:mm:ss.SSSZZ", time zone is ignored.
> Source File:
> TimeColumn
> 03-21-2017T03:30:02Z
> Source code1:
> Dataset dataset = getSqlContext().read()
> .option(DAWBConstant.PARSER_LIB, "commons")
> .option(INFER_SCHEMA, "true")
> .option(DAWBConstant.DELIMITER, ",")
> .option(DAWBConstant.QUOTE, "\"")
> .option(DAWBConstant.ESCAPE, "\\")
> .option("timestampFormat" , "MM-dd-yyyy'T'HH:mm:ss.SSSZZ")
> .option(DAWBConstant.MODE, Mode.PERMISSIVE)
> .csv(sourceFile);
> Result: TimeColumn [ StringType] and value is "03-21-2017T03:30:02Z", but expected result is TimeCoumn should be of "TimestampType" and should consider time zone for manipulation
> Source code2:
> Dataset dataset = getSqlContext().read()
> .option(DAWBConstant.PARSER_LIB, "commons")
> .option(INFER_SCHEMA, "true")
> .option(DAWBConstant.DELIMITER, ",")
> .option(DAWBConstant.QUOTE, "\"")
> .option(DAWBConstant.ESCAPE, "\\")
> .option("timestampFormat" , "MM-dd-yyyy'T'HH:mm:ss")
> .option(DAWBConstant.MODE, Mode.PERMISSIVE)
> .csv(sourceFile);
> Result: TimeColumn [ TimestampType] and value is "2017-04-22 03:30:02.0", but expected result is TimeCoumn should consider time zone for manipulation
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org