You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Murali Aakula (JIRA)" <ji...@apache.org> on 2019/03/04 14:40:00 UTC
[jira] [Comment Edited] (SPARK-26994) Enhance StructField to accept
number format or date format
[ https://issues.apache.org/jira/browse/SPARK-26994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783425#comment-16783425 ]
Murali Aakula edited comment on SPARK-26994 at 3/4/19 2:39 PM:
---------------------------------------------------------------
We could have defined that format conversion in the setter methods. There are multiple alternatives to solve our problem. I'm sure that everyone may have implemented in one or the other alternatives. Also, there will be performance issues while converting form Raw to User defined object (mentioned in the spark documentation).
So I thought that, it will actually make sense, if the parsers/readers (Json, CSV, ..) can handle this conversion. StructField just holds the values(patterns) just like column name and the column datatype. The conversion logic will be in the parsers/readers. Hope this makesense.
Please reopen this enhancement If you agree with me. Thanks.
was (Author: maakula):
We could have defined that format conversion in the setter methods. There are multiple alternatives to solve our problem. I'm sure that everyone may have implemented in one or the other alternatives. Also, there will be performance issues while converting form Raw to User defined object (mentioned in the spark documentation).
So I thought that, it will actually make sense, if the parsers/readers (Json, CSV, ..) can handle this conversion. StructField just holds the values(patterns) just like column name and the column datatype. The conversion logic will be in the parsers/readers. Hope this makesense.
> Enhance StructField to accept number format or date format
> ----------------------------------------------------------
>
> Key: SPARK-26994
> URL: https://issues.apache.org/jira/browse/SPARK-26994
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Murali Aakula
> Priority: Major
>
> Enhance StructField to accept number format or date format and Enahcne reader/steramreader and writer/streamwriter to use these formats.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org