You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "maytasm (via GitHub)" <gi...@apache.org> on 2023/08/29 05:58:46 UTC

[GitHub] [iceberg] maytasm opened a new pull request, #8412: Fix Iceberg handle literal short and byte

maytasm opened a new pull request, #8412:
URL: https://github.com/apache/iceberg/pull/8412

   Spark 3 fail to plan query with filter on short type literal 
   
   i.e. `select * from some_hive_table_with_smallint_column where small_int_column = 1`
   
   where `small_int_column` is a smallint type column. So even if the 1 is cast as int or the `small_int_column` is cast as int, Spark will convert the 1 literal to short type. When this filter (with short type literal) is pushed down to Iceberg, Iceberg throws an Exception saying that Cannot create expression literal from java.lang.Short
   
   This problem is also reproducible when the column is tinyint. In this case, Iceberg throws an Exception saying that Cannot create expression literal from java.lang.Byte
   
   Note that this query works in Spark 2.4 as Spark 2.4 doesn't push down the query to Iceberg.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on pull request #8412: Fix Iceberg to handle literal short and byte

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #8412:
URL: https://github.com/apache/iceberg/pull/8412#issuecomment-1702274726

   > What table format are you using?
   
   Just iceberg format
   
   > I encountered this problem on a Hive table with tinyint / smallint data type.
   
   Does the Hive table here mean an Iceberg table using a Hive catalog?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] ConeyLiu commented on pull request #8412: Fix Iceberg to handle literal short and byte

Posted by "ConeyLiu (via GitHub)" <gi...@apache.org>.
ConeyLiu commented on PR #8412:
URL: https://github.com/apache/iceberg/pull/8412#issuecomment-1702252315

   I tested the master code in Spark 3.4.0 and it works correctly.
   ```scala
   scala> spark.sql("create table local.db.test_short(id short) using iceberg").show()
   ++
   ||
   ++
   ++
   
   scala> spark.sql("insert into local.db.test_short values(1), (2), (3)").show()
   ++
   ||
   ++
   ++
   
   scala> spark.sql("select * from local.db.test_short").show()
   +---+
   | id|
   +---+
   |  1|
   |  2|
   |  3|
   +---+
   
   scala> spark.sql("select * from local.db.test_short where id = 2").show()
   +---+
   | id|
   +---+
   |  2|
   +---+
   ```
   
   It seems Iceberg promotes the short to int:
   ```scala
   scala> spark.sql("desc table local.db.test_short").show()
   +--------+---------+-------+
   |col_name|data_type|comment|
   +--------+---------+-------+
   |      id|      int|   null|
   +--------+---------+-------+
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] maytasm commented on pull request #8412: Fix Iceberg to handle literal short and byte

Posted by "maytasm (via GitHub)" <gi...@apache.org>.
maytasm commented on PR #8412:
URL: https://github.com/apache/iceberg/pull/8412#issuecomment-1702264636

   > I tested the master code in Spark 3.4.0 and it works correctly.
   > 
   > ```scala
   > scala> spark.sql("create table local.db.test_short(id short) using iceberg").show()
   > ++
   > ||
   > ++
   > ++
   > 
   > scala> spark.sql("insert into local.db.test_short values(1), (2), (3)").show()
   > ++
   > ||
   > ++
   > ++
   > 
   > scala> spark.sql("select * from local.db.test_short").show()
   > +---+
   > | id|
   > +---+
   > |  1|
   > |  2|
   > |  3|
   > +---+
   > 
   > scala> spark.sql("select * from local.db.test_short where id = 2").show()
   > +---+
   > | id|
   > +---+
   > |  2|
   > +---+
   > ```
   > 
   > It seems Iceberg promotes the short to int:
   > 
   > ```scala
   > scala> spark.sql("desc table local.db.test_short").show()
   > +--------+---------+-------+
   > |col_name|data_type|comment|
   > +--------+---------+-------+
   > |      id|      int|   null|
   > +--------+---------+-------+
   > ```
   
   What table format are you using? 
   I encountered this problem on a Hive table with tinyint / smallint data type.
   For example, 
   I ran `create table hive.mmonsereenusorn.mmonsereenusorn_test_numbers_3 (a tinyint, b smallint, c int, d bigint, e real, f double, g double precision, h decimal)` in Trino to create the Hive table.
   In spark, this table has tinyint and smallint
   ```
   spark-sql-3.3> desc mmonsereenusorn.mmonsereenusorn_test_numbers_3;
   a                   	tinyint             	                    
   b                   	smallint            	                    
   c                   	int                 	                    
   d                   	bigint              	                    
   e                   	float               	                    
   f                   	double              	                    
   g                   	double              	                    
   h                   	decimal(38,0)       	                    
                       	                    	                    
   # Partitioning      	                    	                    
   Not partitioned     	                    	                    
   Time taken: 4.125 seconds, Fetched 11 row(s)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] maytasm commented on pull request #8412: Fix Iceberg to handle literal short and byte

Posted by "maytasm (via GitHub)" <gi...@apache.org>.
maytasm commented on PR #8412:
URL: https://github.com/apache/iceberg/pull/8412#issuecomment-1702307412

   > > What table format are you using?
   > 
   > Just iceberg format
   > 
   > > I encountered this problem on a Hive table with tinyint / smallint data type.
   > 
   > Does the Hive table here mean an Iceberg table using a Hive catalog?
   
   The table is a Hive table, not an Iceberg table. See the difference at https://www.starburst.io/blog/how-to-migrate-your-hive-tables-to-apache-iceberg/ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix Iceberg to handle literal short and byte [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8412:
URL: https://github.com/apache/iceberg/pull/8412#issuecomment-2350730670

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix Iceberg to handle literal short and byte [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8412: Fix Iceberg to handle literal short and byte
URL: https://github.com/apache/iceberg/pull/8412


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix Iceberg to handle literal short and byte [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8412:
URL: https://github.com/apache/iceberg/pull/8412#issuecomment-2364773558

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org