You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2019/02/15 12:29:42 UTC
[GitHub] prodeezy opened a new issue #99: Iceberg fails to return results
when filtered on complex columns
prodeezy opened a new issue #99: Iceberg fails to return results when filtered on complex columns
URL: https://github.com/apache/incubator-iceberg/issues/99
# Sample JSON Data
scala> json.show
+----+-------+--------------------+
| age| name| friends|
+----+-------+--------------------+
|null|Michael| null|
| 30| Andy|[Josh -> 10, Bria...|
| 19| Justin|[Bharat -> 15, Gau...|
+----+-------+--------------------+
# Json Table Schema
scala> json.printSchema
root
|-- age: integer (nullable = true)
|-- name: string (nullable = true)
|-- friends: map (nullable = true)
| |-- key: string
| |-- value: integer (valueContainsNull = true)
# Predicate works on vanilla spark
scala> spark.sql("select * from people_json_complex where friends['Josh'] = 10").show()
sqlDF: org.apache.spark.sql.DataFrame = [age: int, name: string ... 1 more field]
+---+----+--------------------+
|age|name| friends|
+---+----+--------------------+
| 30|Andy|[Josh -> 10, Bria...|
+---+----+--------------------+
# Sample data in Iceberg format
scala> val iceDf = spark.read.format("iceberg").load("iceberg-people-complex")
scala> iceDf.show()
+----+-------+--------------------+
| age| name| friends|
+----+-------+--------------------+
|null|Michael| null|
| 30| Andy|[Josh -> 10, Bria...|
| 19| Justin|[Bharat -> 15, Xa...|
+----+-------+--------------------+
# Schema on iceberg table
scala> iceDf.printSchema
root
|-- age: integer (nullable = true)
|-- name: string (nullable = true)
|-- friends: map (nullable = true)
| |-- key: string
| |-- value: integer (valueContainsNull = true)
# Create table over iceberg data
scala> iceDf.createOrReplaceTempView("people_iceberg_complex")
# Filter on Complex (map) column fails to return result
scala> spark.sql("select * from people_iceberg_complex where friends['Josh'] = 10").show()
sqlDF: org.apache.spark.sql.DataFrame = [age: int, name: string ... 1 more field]
+---+----+-------+
|age|name|friends|
+---+----+-------+
+---+----+-------+
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org