You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Gabor Somogyi (JIRA)" <ji...@apache.org> on 2019/02/07 19:50:00 UTC
[jira] [Updated] (SPARK-26845) Avro from_avro to_avro roundtrip
fails if data type is string
[ https://issues.apache.org/jira/browse/SPARK-26845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabor Somogyi updated SPARK-26845:
----------------------------------
Description:
I was playing with AvroFunctionsSuite and creates a situation where test fails which I believe it shouldn't:
{code:java}
test("roundtrip in to_avro and from_avro - string") {
val df = spark.createDataset(Seq("1", "2", "3")).select('value.cast("string").as("str"))
val avroDF = df.select(to_avro('str).as("b"))
val avroTypeStr = s"""
|{
| "type": "string",
| "name": "str"
|}
""".stripMargin
checkAnswer(avroDF.select(from_avro('b, avroTypeStr)), df)
}
{code}
{code:java}
== Results ==
!== Correct Answer - 3 == == Spark Answer - 3 ==
!struct<str:string> struct<from_avro(b):string>
![1] []
![2] []
![3] []
{code}
was:
I was playing with AvroFunctionsSuite and creates a situation where test fails which I believe it shouldn't:
{code:java}
test("roundtrip in to_avro and from_avro - string") {
val df = spark.createDataset(Seq("1", "2", "3")).select('value.cast("string").as("str"))
val avroDF = df.select(to_avro('str).as("b"))
val avroTypeStr = s"""
|{
| "type": "string",
| "name": "str"
|}
""".stripMargin
checkAnswer(avroDF.select(from_avro('b, avroTypeStr)), df)
}
{code}
> Avro from_avro to_avro roundtrip fails if data type is string
> -------------------------------------------------------------
>
> Key: SPARK-26845
> URL: https://issues.apache.org/jira/browse/SPARK-26845
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0, 3.0.0
> Reporter: Gabor Somogyi
> Priority: Critical
>
> I was playing with AvroFunctionsSuite and creates a situation where test fails which I believe it shouldn't:
> {code:java}
> test("roundtrip in to_avro and from_avro - string") {
> val df = spark.createDataset(Seq("1", "2", "3")).select('value.cast("string").as("str"))
> val avroDF = df.select(to_avro('str).as("b"))
> val avroTypeStr = s"""
> |{
> | "type": "string",
> | "name": "str"
> |}
> """.stripMargin
> checkAnswer(avroDF.select(from_avro('b, avroTypeStr)), df)
> }
> {code}
> {code:java}
> == Results ==
> !== Correct Answer - 3 == == Spark Answer - 3 ==
> !struct<str:string> struct<from_avro(b):string>
> ![1] []
> ![2] []
> ![3] []
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org