[jira] [Created] (LIVY-455) Livy Spark SQL "not a primitive class java.math.BigDecimal" when querying decimal columns

["Carter Shanklin (JIRA)" <ji...@apache.org>]

Carter Shanklin created LIVY-455:
------------------------------------

             Summary: Livy Spark SQL "not a primitive class java.math.BigDecimal" when querying decimal columns
                 Key: LIVY-455
                 URL: https://issues.apache.org/jira/browse/LIVY-455
             Project: Livy
          Issue Type: Bug
            Reporter: Carter Shanklin


I was using the new SQL option introduced by LIVY-19 and ran into what appears to be a limitation with DECIMAL support.

The set of queries I run roughly looks like this:

{code}
drop table if exists sales_fact_1998;
create external table sales_fact_1998(
    product_id int,time_id int,customer_id int,promotion_id int,
    store_id int,store_sales decimal(10,4),store_cost decimal(10,4),
    unit_sales decimal(10,4))
stored as parquet
location '${root}/sales_fact_1998';

select
    customer_id,
    sum(store_sales) - sum(store_cost) gross_profit
from
    sales_fact_1998
where
    store_id=${store_id}
    and time_id=${time_id}
group by
    customer_id
order by
    gross_profit desc
limit 10;
{code}

The full response I get from Livy is:
{code}
{"id":2,"code":"\n\nselect\n    customer_id,\n    sum(store_sales) - sum(store_cost) gross_profit\nfrom\n    sales_fact_1998\nwhere\n    store_id=12\n    and time_id=1037\ngroup by\n    customer_id\norder by\n    gross_profit desc\nlimit 10","state":"available","output":{"status":"error","execution_count":2,"ename":"Error","evalue":"not a primitive class java.math.BigDecimal","traceback":["scala.sys.package$.error(package.scala:27)","org.apache.livy.shaded.json4s.Extraction$.writePrimitive(Extraction.scala:216)","org.apache.livy.shaded.json4s.Extraction$.internalDecomposeWithBuilder(Extraction.scala:127)","org.apache.livy.shaded.json4s.Extraction$.internalDecomposeWithBuilder(Extraction.scala:141)","org.apache.livy.shaded.json4s.Extraction$.internalDecomposeWithBuilder(Extraction.scala:151)","org.apache.livy.shaded.json4s.Extraction$.decomposeWithBuilder(Extraction.scala:67)","org.apache.livy.shaded.json4s.Extraction$.decompose(Extraction.scala:194)","org.apache.livy.repl.SQLInterpreter.execute(SQLInterpreter.scala:101)","org.apache.livy.repl.Session$$anonfun$7.apply(Session.scala:274)","org.apache.livy.repl.Session$$anonfun$7.apply(Session.scala:272)","scala.Option.map(Option.scala:146)","org.apache.livy.repl.Session.org$apache$livy$repl$Session$$executeCode(Session.scala:272)","org.apache.livy.repl.Session$$anonfun$execute$1.apply$mcV$sp(Session.scala:168)","org.apache.livy.repl.Session$$anonfun$execute$1.apply(Session.scala:163)","org.apache.livy.repl.Session$$anonfun$execute$1.apply(Session.scala:163)","scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)","scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)","java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)","java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)","java.lang.Thread.run(Thread.java:748)"]},"progress":1.0}
{code}

My REST calls are:
{code}
POST {} to http://192.168.59.21:9999/sessions

POST {"code": "drop table if exists sales_fact_1998", "kind": "sql"} to http://192.168.59.21:9999/sessions/1/statements

POST {"code": "\ncreate external table sales_fact_1998(\n    product_id int,time_id int,customer_id int,promotion_id int,\n    store_id int,store_sales decimal(10,4),store_cost decimal(10,4),\n    unit_sales decimal(10,4))\nstored as parquet\nlocation '/apps/hive/warehouse/foodmart_parquet.db/sales_fact_1998'", "kind": "sql"} to http://192.168.59.21:9999/sessions/1/statements

POST {"code": "\n\nselect\n    customer_id,\n    sum(store_sales) - sum(store_cost) gross_profit\nfrom\n    sales_fact_1998\nwhere\n    store_id=12\n    and time_id=1037\ngroup by\n    customer_id\norder by\n    gross_profit desc\nlimit 10", "kind": "sql"} to http://192.168.59.21:9999/sessions/1/statements
{code}

If I change the last statement to say cast(sum(store_sales) - sum(store_cost) as double) everything works.

If anyone wants the Parquet file, find it at https://github.com/cartershanklin/structor/tree/master/modules/sample_hive_data/files/foodmart_parquet.db/sales_fact_1998



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)