You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Li Yuanjian (JIRA)" <ji...@apache.org> on 2018/08/16 00:52:00 UTC
[jira] [Updated] (SPARK-25104) Validate user specified output
schema
[ https://issues.apache.org/jira/browse/SPARK-25104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Li Yuanjian updated SPARK-25104:
--------------------------------
Description:
With code changes in [https://github.com/apache/spark/pull/21847|https://github.com/apache/spark/pull/21847] , Spark can write out data as per user provided output schema.
To make it more robust and user friendly, we should validate the Avro schema before tasks launched.
Also we should support output logical decimal type as BYTES (By default we output as FIXED)
was:
With code changes in [https://github.com/apache/spark/pull/21847|https://github.com/apache/spark/pull/21847,] , Spark can write out data as per user provided output schema.
To make it more robust and user friendly, we should validate the Avro schema before tasks launched.
Also we should support output logical decimal type as BYTES (By default we output as FIXED)
> Validate user specified output schema
> -------------------------------------
>
> Key: SPARK-25104
> URL: https://issues.apache.org/jira/browse/SPARK-25104
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Gengliang Wang
> Assignee: Gengliang Wang
> Priority: Major
> Fix For: 2.4.0
>
>
> With code changes in [https://github.com/apache/spark/pull/21847|https://github.com/apache/spark/pull/21847] , Spark can write out data as per user provided output schema.
> To make it more robust and user friendly, we should validate the Avro schema before tasks launched.
> Also we should support output logical decimal type as BYTES (By default we output as FIXED)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org