You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Tae-kyeom, Kim (Jira)" <ji...@apache.org> on 2020/03/13 00:48:00 UTC
[jira] [Created] (SPARK-31139) Fileformat datasources (ORC, Json)
case sensitivity regressions
Tae-kyeom, Kim created SPARK-31139:
--------------------------------------
Summary: Fileformat datasources (ORC, Json) case sensitivity regressions
Key: SPARK-31139
URL: https://issues.apache.org/jira/browse/SPARK-31139
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Reporter: Tae-kyeom, Kim
In addition to https://issues.apache.org/jira/browse/SPARK-31116
Not only parquet, json and orc also have case sensitivity issues.
Following demonstrate test failure based SPARK-31116's test cases.
----
{code:java}
[info] - SPARK-31116: Select simple columns correctly in case insensitive manner *** FAILED *** (4 seconds, 277 milliseconds) [info] Results do not match for query: [info] Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]] [info] Timezone Env: [info] [info] == Parsed Logical Plan == [info] Relation[camelcase#56] json [info] [info] == Analyzed Logical Plan == [info] camelcase: string [info] Relation[camelcase#56] json [info] [info] == Optimized Logical Plan == [info] Relation[camelcase#56] json [info] [info] == Physical Plan == [info] FileScan json [camelcase#56] Batched: false, DataFilters: [], Format: JSON, Location: InMemoryFileIndex[file:/Users/kimtkyeom/Dev/spark_devel/target/tmp/spark-95f1357a-85c9-444f-bdcc-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<camelcase:string> [info] [info] == Results == [info] [info] == Results == [info] !== Correct Answer - 1 == == Spark Answer - 1 == [info] !struct<> struct<camelcase:string> [info] ![A] [null] (QueryTest.scala:248)
[info] - SPARK-31116: Select nested columns correctly in case insensitive manner *** FAILED *** (2 seconds, 117 milliseconds) [info] Results do not match for query: [info] Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]] [info] Timezone Env: [info] [info] == Parsed Logical Plan == [info] Relation[StructColumn#147] json [info] [info] == Analyzed Logical Plan == [info] StructColumn: struct<LowerCase:bigint,camelcase:bigint> [info] Relation[StructColumn#147] json [info] [info] == Optimized Logical Plan == [info] Relation[StructColumn#147] json [info] [info] == Physical Plan == [info] FileScan json [StructColumn#147] Batched: false, DataFilters: [], Format: JSON, Location: InMemoryFileIndex[file:/Users/kimtkyeom/Dev/spark_devel/target/tmp/spark-f9ecd1a4-e5aa-4dd7-bdfd-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<StructColumn:struct<LowerCase:bigint,camelcase:bigint>> [info] [info] == Results == [info] [info] == Results == [info] !== Correct Answer - 1 == == Spark Answer - 1 == [info] !struct<> struct<StructColumn:struct<LowerCase:bigint,camelcase:bigint>> [info] ![[0,1]] [[null,null]] (QueryTest.scala:248)
[info] - SPARK-31116: Select nested columns correctly in case sensitive manner *** FAILED *** (871 milliseconds) [info] Results do not match for query: [info] Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]] [info] Timezone Env: [info] [info] == Parsed Logical Plan == [info] Relation[StructColumn#329] json [info] [info] == Analyzed Logical Plan == [info] StructColumn: struct<LowerCase:bigint,camelcase:bigint> [info] Relation[StructColumn#329] json [info] [info] == Optimized Logical Plan == [info] Relation[StructColumn#329] json [info] [info] == Physical Plan == [info] FileScan json [StructColumn#329] Batched: false, DataFilters: [], Format: JSON, Location: InMemoryFileIndex[file:/Users/kimtkyeom/Dev/spark_devel/target/tmp/spark-612baf76-a9d0-41e5-89f4-..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<StructColumn:struct<LowerCase:bigint,camelcase:bigint>> [info] [info] == Results == [info] [info] == Results == [info] !== Correct Answer - 1 == == Spark Answer - 1 == [info] !struct<> struct<StructColumn:struct<LowerCase:bigint,camelcase:bigint>> [info] ![null] [[null,null]] (QueryTest.scala:248)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org