You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "JESSE CHEN (JIRA)" <ji...@apache.org> on 2016/03/23 18:52:25 UTC
[jira] [Updated] (SPARK-14096) SPARK-SQL CLI returns NPE
[ https://issues.apache.org/jira/browse/SPARK-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
JESSE CHEN updated SPARK-14096:
-------------------------------
Affects Version/s: (was: 1.6.0)
2.0.0
Description:
Trying to run TPCDS query 06 in spark-sql shell received the following error in the middle of a stage; but running another query 38 succeeded:
NPE:
{noformat}
16/03/22 15:12:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
16/03/22 15:12:56 INFO scheduler.TaskSetManager: Finished task 65.0 in stage 10.0 (TID 622) in 171 ms on localhost (30/200)
16/03/22 15:12:56 ERROR scheduler.TaskResultGetter: Exception while getting task result
com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
Serialization trace:
underlying (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:626)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
at com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)
at com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
at org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)
at org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)
at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)
at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)
at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1790)
at org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)
at org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)
at scala.math.Ordering$$anon$4.compare(Ordering.scala:111)
at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:669)
at java.util.PriorityQueue.siftUp(PriorityQueue.java:645)
at java.util.PriorityQueue.offer(PriorityQueue.java:344)
at java.util.PriorityQueue.add(PriorityQueue.java:321)
at com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)
at com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:651)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605)
... 15 more
16/03/22 15:12:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
16/03/22 15:12:56 INFO scheduler.TaskSetManager: Finished task 66.0 in stage 10.0 (TID 623) in 171 ms on localhost (31/200)
16/03/22 15:12:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
{noformat}
query 06 (caused the above NPE):
{noformat}
select a.ca_state state, count(*) cnt
from customer_address a
join customer c on a.ca_address_sk = c.c_current_addr_sk
join store_sales s on c.c_customer_sk = s.ss_customer_sk
join date_dim d on s.ss_sold_date_sk = d.d_date_sk
join item i on s.ss_item_sk = i.i_item_sk
join (select distinct d_month_seq
from date_dim
where d_year = 2001
and d_moy = 1 ) tmp1 ON d.d_month_seq = tmp1.d_month_seq
join
(select j.i_category, avg(j.i_current_price) as avg_i_current_price
from item j group by j.i_category) tmp2 on tmp2.i_category = i.i_category
where
i.i_current_price > 1.2 * tmp2.avg_i_current_price
group by a.ca_state
having count(*) >= 10
order by cnt
limit 100;
{noformat}
query 38 (succeeded)
{noformat}
select count(*) from (
select distinct c_last_name, c_first_name, d_date
from store_sales, date_dim, customer
where store_sales.ss_sold_date_sk = date_dim.d_date_sk
and store_sales.ss_customer_sk = customer.c_customer_sk
and d_month_seq between 1200 and 1200 + 11
intersect
select distinct c_last_name, c_first_name, d_date
from catalog_sales, date_dim, customer
where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
and d_month_seq between 1200 and 1200 + 11
intersect
select distinct c_last_name, c_first_name, d_date
from web_sales, date_dim, customer
where web_sales.ws_sold_date_sk = date_dim.d_date_sk
and web_sales.ws_bill_customer_sk = customer.c_customer_sk
and d_month_seq between 1200 and 1200 + 11
) hot_cust
limit 100;
{noformat}
was:
Testing Spark SQL using TPC queries. Query 49 returns wrong results compared to official result set. This is at 1GB SF (validation run).
SparkSQL has right answer but in wrong order (and there is an 'order by' in the query).
Actual results:
{noformat}
store,9797,0.80000000000000000000,2,2]
[store,12641,0.81609195402298850575,3,3]
[store,6661,0.92207792207792207792,7,7]
[store,13013,0.94202898550724637681,8,8]
[store,9029,1.00000000000000000000,10,10]
[web,15597,0.66197183098591549296,3,3]
[store,14925,0.96470588235294117647,9,9]
[store,4063,1.00000000000000000000,10,10]
[catalog,8929,0.76250000000000000000,7,7]
[store,11589,0.82653061224489795918,6,6]
[store,1171,0.82417582417582417582,5,5]
[store,9471,0.77500000000000000000,1,1]
[catalog,12577,0.65591397849462365591,3,3]
[web,97,0.90361445783132530120,9,8]
[web,85,0.85714285714285714286,8,7]
[catalog,361,0.74647887323943661972,5,5]
[web,2915,0.69863013698630136986,4,4]
[web,117,0.92500000000000000000,10,9]
[catalog,9295,0.77894736842105263158,9,9]
[web,3305,0.73750000000000000000,6,16]
[catalog,16215,0.79069767441860465116,10,10]
[web,7539,0.59000000000000000000,1,1]
[catalog,17543,0.57142857142857142857,1,1]
[catalog,3411,0.71641791044776119403,4,4]
[web,11933,0.71717171717171717172,5,5]
[catalog,14513,0.63541666666666666667,2,2]
[store,15839,0.81632653061224489796,4,4]
[web,3337,0.62650602409638554217,2,2]
[web,5299,0.92708333333333333333,11,10]
[catalog,8189,0.74698795180722891566,6,6]
[catalog,14869,0.77173913043478260870,8,8]
[web,483,0.80000000000000000000,7,6]
{noformat}
Expected results:
{noformat}
+---------+-------+--------------------+-------------+---------------+
| CHANNEL | ITEM | RETURN_RATIO | RETURN_RANK | CURRENCY_RANK |
+---------+-------+--------------------+-------------+---------------+
| catalog | 17543 | .5714285714285714 | 1 | 1 |
| catalog | 14513 | .6354166666666666 | 2 | 2 |
| catalog | 12577 | .6559139784946236 | 3 | 3 |
| catalog | 3411 | .7164179104477611 | 4 | 4 |
| catalog | 361 | .7464788732394366 | 5 | 5 |
| catalog | 8189 | .7469879518072289 | 6 | 6 |
| catalog | 8929 | .7625000000000000 | 7 | 7 |
| catalog | 14869 | .7717391304347826 | 8 | 8 |
| catalog | 9295 | .7789473684210526 | 9 | 9 |
| catalog | 16215 | .7906976744186046 | 10 | 10 |
| store | 9471 | .7750000000000000 | 1 | 1 |
| store | 9797 | .8000000000000000 | 2 | 2 |
| store | 12641 | .8160919540229885 | 3 | 3 |
| store | 15839 | .8163265306122448 | 4 | 4 |
| store | 1171 | .8241758241758241 | 5 | 5 |
| store | 11589 | .8265306122448979 | 6 | 6 |
| store | 6661 | .9220779220779220 | 7 | 7 |
| store | 13013 | .9420289855072463 | 8 | 8 |
| store | 14925 | .9647058823529411 | 9 | 9 |
| store | 4063 | 1.0000000000000000 | 10 | 10 |
| store | 9029 | 1.0000000000000000 | 10 | 10 |
| web | 7539 | .5900000000000000 | 1 | 1 |
| web | 3337 | .6265060240963855 | 2 | 2 |
| web | 15597 | .6619718309859154 | 3 | 3 |
| web | 2915 | .6986301369863013 | 4 | 4 |
| web | 11933 | .7171717171717171 | 5 | 5 |
| web | 3305 | .7375000000000000 | 6 | 16 |
| web | 483 | .8000000000000000 | 7 | 6 |
| web | 85 | .8571428571428571 | 8 | 7 |
| web | 97 | .9036144578313253 | 9 | 8 |
| web | 117 | .9250000000000000 | 10 | 9 |
| web | 5299 | .9270833333333333 | 11 | 10 |
+---------+-------+--------------------+-------------+---------------+
{noformat}
Query used:
{noformat}
-- start query 49 in stream 0 using template query49.tpl and seed QUALIFICATION
select
'web' as channel
,web.item
,web.return_ratio
,web.return_rank
,web.currency_rank
from (
select
item
,return_ratio
,currency_ratio
,rank() over (order by return_ratio) as return_rank
,rank() over (order by currency_ratio) as currency_rank
from
( select ws.ws_item_sk as item
,(cast(sum(coalesce(wr.wr_return_quantity,0)) as decimal(15,4))/
cast(sum(coalesce(ws.ws_quantity,0)) as decimal(15,4) )) as return_ratio
,(cast(sum(coalesce(wr.wr_return_amt,0)) as decimal(15,4))/
cast(sum(coalesce(ws.ws_net_paid,0)) as decimal(15,4) )) as currency_ratio
from
web_sales ws left outer join web_returns wr
on (ws.ws_order_number = wr.wr_order_number and
ws.ws_item_sk = wr.wr_item_sk)
,date_dim
where
wr.wr_return_amt > 10000
and ws.ws_net_profit > 1
and ws.ws_net_paid > 0
and ws.ws_quantity > 0
and ws_sold_date_sk = d_date_sk
and d_year = 2001
and d_moy = 12
group by ws.ws_item_sk
) in_web
) web
where
(
web.return_rank <= 10
or
web.currency_rank <= 10
)
union
select
'catalog' as channel
,catalog.item
,catalog.return_ratio
,catalog.return_rank
,catalog.currency_rank
from (
select
item
,return_ratio
,currency_ratio
,rank() over (order by return_ratio) as return_rank
,rank() over (order by currency_ratio) as currency_rank
from
( select
cs.cs_item_sk as item
,(cast(sum(coalesce(cr.cr_return_quantity,0)) as decimal(15,4))/
cast(sum(coalesce(cs.cs_quantity,0)) as decimal(15,4) )) as return_ratio
,(cast(sum(coalesce(cr.cr_return_amount,0)) as decimal(15,4))/
cast(sum(coalesce(cs.cs_net_paid,0)) as decimal(15,4) )) as currency_ratio
from
catalog_sales cs left outer join catalog_returns cr
on (cs.cs_order_number = cr.cr_order_number and
cs.cs_item_sk = cr.cr_item_sk)
,date_dim
where
cr.cr_return_amount > 10000
and cs.cs_net_profit > 1
and cs.cs_net_paid > 0
and cs.cs_quantity > 0
and cs_sold_date_sk = d_date_sk
and d_year = 2001
and d_moy = 12
group by cs.cs_item_sk
) in_cat
) catalog
where
(
catalog.return_rank <= 10
or
catalog.currency_rank <=10
)
union
select
'store' as channel
,store.item
,store.return_ratio
,store.return_rank
,store.currency_rank
from (
select
item
,return_ratio
,currency_ratio
,rank() over (order by return_ratio) as return_rank
,rank() over (order by currency_ratio) as currency_rank
from
( select sts.ss_item_sk as item
,(cast(sum(coalesce(sr.sr_return_quantity,0)) as decimal(15,4))/cast(sum(coalesce(sts.ss_quantity,0)) as decimal(15,4) )) as return_ratio
,(cast(sum(coalesce(sr.sr_return_amt,0)) as decimal(15,4))/cast(sum(coalesce(sts.ss_net_paid,0)) as decimal(15,4) )) as currency_ratio
from
store_sales sts left outer join store_returns sr
on (sts.ss_ticket_number = sr.sr_ticket_number and sts.ss_item_sk = sr.sr_item_sk)
,date_dim
where
sr.sr_return_amt > 10000
and sts.ss_net_profit > 1
and sts.ss_net_paid > 0
and sts.ss_quantity > 0
and ss_sold_date_sk = d_date_sk
and d_year = 2001
and d_moy = 12
group by sts.ss_item_sk
) in_store
) store
where (
store.return_rank <= 10
or
store.currency_rank <= 10
)
order by 1,4,5
limit 100;
-- end query 49 in stream 0 using template query49.tpl
{noformat}
> SPARK-SQL CLI returns NPE
> -------------------------
>
> Key: SPARK-14096
> URL: https://issues.apache.org/jira/browse/SPARK-14096
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: JESSE CHEN
> Labels: tpcds-result-mismatch
>
> Trying to run TPCDS query 06 in spark-sql shell received the following error in the middle of a stage; but running another query 38 succeeded:
> NPE:
> {noformat}
> 16/03/22 15:12:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
> 16/03/22 15:12:56 INFO scheduler.TaskSetManager: Finished task 65.0 in stage 10.0 (TID 622) in 171 ms on localhost (30/200)
> 16/03/22 15:12:56 ERROR scheduler.TaskResultGetter: Exception while getting task result
> com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
> Serialization trace:
> underlying (org.apache.spark.util.BoundedPriorityQueue)
> at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:626)
> at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
> at com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:25)
> at com.twitter.chill.SomeSerializer.read(SomeSerializer.scala:19)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
> at org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:312)
> at org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:87)
> at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:66)
> at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)
> at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:57)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1790)
> at org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:56)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:157)
> at org.apache.spark.sql.catalyst.expressions.codegen.LazilyGeneratedOrdering.compare(GenerateOrdering.scala:148)
> at scala.math.Ordering$$anon$4.compare(Ordering.scala:111)
> at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:669)
> at java.util.PriorityQueue.siftUp(PriorityQueue.java:645)
> at java.util.PriorityQueue.offer(PriorityQueue.java:344)
> at java.util.PriorityQueue.add(PriorityQueue.java:321)
> at com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:78)
> at com.twitter.chill.java.PriorityQueueSerializer.read(PriorityQueueSerializer.java:31)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:651)
> at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605)
> ... 15 more
> 16/03/22 15:12:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
> 16/03/22 15:12:56 INFO scheduler.TaskSetManager: Finished task 66.0 in stage 10.0 (TID 623) in 171 ms on localhost (31/200)
> 16/03/22 15:12:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
> {noformat}
> query 06 (caused the above NPE):
> {noformat}
> select a.ca_state state, count(*) cnt
> from customer_address a
> join customer c on a.ca_address_sk = c.c_current_addr_sk
> join store_sales s on c.c_customer_sk = s.ss_customer_sk
> join date_dim d on s.ss_sold_date_sk = d.d_date_sk
> join item i on s.ss_item_sk = i.i_item_sk
> join (select distinct d_month_seq
> from date_dim
> where d_year = 2001
> and d_moy = 1 ) tmp1 ON d.d_month_seq = tmp1.d_month_seq
> join
> (select j.i_category, avg(j.i_current_price) as avg_i_current_price
> from item j group by j.i_category) tmp2 on tmp2.i_category = i.i_category
> where
> i.i_current_price > 1.2 * tmp2.avg_i_current_price
> group by a.ca_state
> having count(*) >= 10
> order by cnt
> limit 100;
> {noformat}
> query 38 (succeeded)
> {noformat}
> select count(*) from (
> select distinct c_last_name, c_first_name, d_date
> from store_sales, date_dim, customer
> where store_sales.ss_sold_date_sk = date_dim.d_date_sk
> and store_sales.ss_customer_sk = customer.c_customer_sk
> and d_month_seq between 1200 and 1200 + 11
> intersect
> select distinct c_last_name, c_first_name, d_date
> from catalog_sales, date_dim, customer
> where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
> and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
> and d_month_seq between 1200 and 1200 + 11
> intersect
> select distinct c_last_name, c_first_name, d_date
> from web_sales, date_dim, customer
> where web_sales.ws_sold_date_sk = date_dim.d_date_sk
> and web_sales.ws_bill_customer_sk = customer.c_customer_sk
> and d_month_seq between 1200 and 1200 + 11
> ) hot_cust
> limit 100;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org