You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by "Dongkyu Hwangbo (JIRA)" <ji...@apache.org> on 2015/08/18 08:16:45 UTC
[jira] [Created] (TAJO-1781) Large-scale overhead is occured
Dongkyu Hwangbo created TAJO-1781:
-------------------------------------
Summary: Large-scale overhead is occured
Key: TAJO-1781
URL: https://issues.apache.org/jira/browse/TAJO-1781
Project: Tajo
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Dongkyu Hwangbo
Priority: Minor
This is TPC-DS q13.
{code:SQL}
select avg(ss_quantity)
,avg(ss_ext_sales_price)
,avg(ss_ext_wholesale_cost)
,sum(ss_ext_wholesale_cost)
from store_sales
,store
,customer_demographics
,household_demographics
,customer_address
,date_dim
where s_store_sk = ss_store_sk
and ss_sold_date_sk = d_date_sk and d_year = 2001
and((ss_hdemo_sk=hd_demo_sk
and cd_demo_sk = ss_cdemo_sk
and cd_marital_status = 'M'
and cd_education_status = 'Advanced Degree'
and ss_sales_price between 100.00 and 150.00
and hd_dep_count = 3
)or
(ss_hdemo_sk=hd_demo_sk
and cd_demo_sk = ss_cdemo_sk
and cd_marital_status = 'S'
and cd_education_status = 'College'
and ss_sales_price between 50.00 and 100.00
and hd_dep_count = 1
) or
(ss_hdemo_sk=hd_demo_sk
and cd_demo_sk = ss_cdemo_sk
and cd_marital_status = 'W'
and cd_education_status = '2 yr Degree'
and ss_sales_price between 150.00 and 200.00
and hd_dep_count = 1
))
and((ss_addr_sk = ca_address_sk
and ca_country = 'United States'
and ca_state in ('TX', 'OH', 'TX')
and ss_net_profit between 100 and 200
) or
(ss_addr_sk = ca_address_sk
and ca_country = 'United States'
and ca_state in ('OR', 'NM', 'KY')
and ss_net_profit between 150 and 300
) or
(ss_addr_sk = ca_address_sk
and ca_country = 'United States'
and ca_state in ('VA', 'TX', 'MS')
and ss_net_profit between 50 and 250
));
{code}
after (almost) 10 minutes, this error log is written. (in worker log)
{noformat}
2015-08-18 14:29:41,220 INFO org.apache.tajo.worker.TaskImpl: One local chunk is added to listTablets
2015-08-18 14:29:41,221 INFO org.apache.tajo.worker.TaskImpl: One local chunk is added to listTablets
2015-08-18 14:40:26,950 INFO org.apache.tajo.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2309ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=1988ms
GC pool 'PS Scavenge' had collection(s): count=1 time=3ms
2015-08-18 15:02:56,295 ERROR org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec: 장치에 남은 공간이 없음
java.io.IOException: 장치에 남은 공간이 없음
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
at org.apache.tajo.storage.RawFile$RawFileAppender.flushBuffer(RawFile.java:532)
at org.apache.tajo.storage.RawFile$RawFileAppender.addTuple(RawFile.java:640)
at org.apache.tajo.storage.HashShuffleAppender.addTuples(HashShuffleAppender.java:92)
at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:117)
at org.apache.tajo.worker.TaskImpl.run(TaskImpl.java:403)
at org.apache.tajo.worker.TaskContainer.run(TaskContainer.java:65)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-08-18 15:02:56,347 ERROR org.apache.tajo.worker.TaskImpl: java.io.IOException: 장치에 남은 공간이 없음
java.io.IOException: java.io.IOException: 장치에 남은 공간이 없음
at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:147)
at org.apache.tajo.worker.TaskImpl.run(TaskImpl.java:403)
at org.apache.tajo.worker.TaskContainer.run(TaskContainer.java:65)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: 장치에 남은 공간이 없음
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
at org.apache.tajo.storage.RawFile$RawFileAppender.flushBuffer(RawFile.java:532)
at org.apache.tajo.storage.RawFile$RawFileAppender.addTuple(RawFile.java:640)
at org.apache.tajo.storage.HashShuffleAppender.addTuples(HashShuffleAppender.java:92)
at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.next(HashShuffleFileWriteExec.java:117)
... 7 more
2015-08-18 15:02:56,348 INFO org.apache.tajo.worker.TaskAttemptContext: Query status of ta_1439875670149_0001_000007_000000_00 is changed to TA_FAILED
2015-08-18 15:02:56,411 INFO org.apache.tajo.worker.TaskImpl: ta_1439875670149_0001_000007_000000_00 completed. Worker's task counter - total:1, succeeded: 0, killed: 0, failed: 1
2015-08-18 15:02:56,413 ERROR org.apache.tajo.querymaster.TaskAttempt: ta_1439875670149_0001_000007_000000_00 FROM dkhwangbo >> java.io.IOException: 장치에 남은 공간이 없음
2015-08-18 15:02:56,413 INFO org.apache.tajo.querymaster.Task: ====================================================================================
2015-08-18 15:02:56,413 INFO org.apache.tajo.querymaster.Task: >>> Task Failed: ta_1439875670149_0001_000007_000000_00, retry:true, attempts:1 <<<
2015-08-18 15:02:56,413 INFO org.apache.tajo.querymaster.Task: ====================================================================================
{noformat}
I think tajo run with inefficient way to run this query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)