You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by "sunfl@certusnet.com.cn" <su...@certusnet.com.cn> on 2015/01/29 03:44:17 UTC
PhoenixIOException : Task rejected from org.apache.phoenix.job.JobManager
Hi, all
I got strange exception hints when doing query like SELECT COUNT(*) FROM SOME_TABLE; through sqlline and the info is
java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException: Task org.apache.phoenix.job.JobManager$JobFutureTask@f55dfff rejected from
org.apache.phoenix.job.JobManager$1@165e96e[Running, pool size = 64, active threads = 64, queued tasks = 5000, completed tasks = 5162]
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
at sqlline.SqlLine.print(SqlLine.java:1735)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
at sqlline.SqlLine.dispatch(SqlLine.java:821)
at sqlline.SqlLine.begin(SqlLine.java:699)
at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
at sqlline.SqlLine.main(SqlLine.java:424)
However, such query like SELECT * FROM SOME_TABLE LIMIT 10 works. Also, other table count query works too. Don't know what causes this kind of issue. Any
available hints are appreciated.
Thanks,
Sun.
CertusNet
Re: Re: PhoenixIOException : Task rejected from org.apache.phoenix.job.JobManager
Posted by "sunfl@certusnet.com.cn" <su...@certusnet.com.cn>.
Hi, James & Jan
Thanks for your detaily illustration for this. Now I can see the balance of tuning both client
and server configs. Actually the table being count queried is so special that stores far more
data than other tables. I did try to increase both configs of "phoenix.query.threadPoolSize"
and "phoenix.query.queueSize" following 2x rate, but that did not work as expected. Then I
increasing phoenix.query.queueSize significantly from default to more larger like 50000,
I can get the query results at last. Maybe that is the cause as the specific table is so special.
I would investigate more deep for what James suggests about chunk size and do UPDATE
STATISTICS for the table.
Thanks so much,
Sun.
CertusNet
From: James Taylor
Date: 2015-01-30 03:38
To: user
Subject: Re: PhoenixIOException : Task rejected from org.apache.phoenix.job.JobManager
Just to add to what Jan said, if work is being rejected by the thread
pool, you can increase your phoenix.query.queueSize. However, you
don't want it to be so high that the client machine will never be able
to complete the queued tasks. By throwing a rejected exception, the
client is being told "you're giving me too much work to do and I may
never finish it". So there's a balance here.
The other side of the equation is how big you want the work to be
chunked up. This corresponds to how much data will be scanned by each
thread. The bigger the amount of work, the less number of chunks of
work will be queued in the client thread pool. The downside of this is
that you decrease the level of parallelism, thus increasing the
latency you'll see to get query results.
The "chunk size" is controlled through the
phoenix.stats.guidepost.per.region or phoenix.stats.guidepost.width
config properties. You'd set one or the other. Setting
phoenix.stats.guidepost.per.region would give you a "relative" chunk
size depending on how the HBase table is configured. If you set it to
1, then you'd only get a single thread scanning to entire region of a
table. If you set it to 2, then you'd get at most two threads scanning
a region. Setting phoenix.stats.guidepost.width on the other hand
gives you an absolute number of bytes that each scan will perform.
Both these config parameters are server-side settings, so you'd need
to set them in the hbase-sites.xml on each region server, bounce your
cluster, and run UPDATE STATISTICS <table name> to have them take
affect.
Hope this helps. These parameters are documented here:
http://phoenix.apache.org/tuning.html and there's some more
information here: http://phoenix.apache.org/update_statistics.html
Thanks,
James
On Thu, Jan 29, 2015 at 9:01 AM, Jan Fernando <jf...@salesforce.com> wrote:
> Hi Sun,
>
> I believe this means you have exhausted the thread pool and the queue that
> buffers the tasks to service Phoenix queries. Each query will have different
> characteristics in terms of the level of parallelism that Phoenix will use
> to service that query. For example some queries require few (or 1) tasks,
> such as a query with LIMIT 10, other queries that hit many regions and
> guideposts require a higher number of tasks. As you use up the queue and get
> near the max some queries may run if they are enough slots left in the queue
> to service it. So queries requiring a low number of slots like a LIMIT 10
> query might run but others won't.
>
> Hope this helps.
>
> --Jan
>
> On Wed, Jan 28, 2015 at 6:44 PM, sunfl@certusnet.com.cn
> <su...@certusnet.com.cn> wrote:
>>
>> Hi, all
>>
>> I got strange exception hints when doing query like SELECT COUNT(*) FROM
>> SOME_TABLE; through sqlline and the info is
>>
>> java.lang.RuntimeException:
>> org.apache.phoenix.exception.PhoenixIOException: Task
>> org.apache.phoenix.job.JobManager$JobFutureTask@f55dfff rejected from
>>
>> org.apache.phoenix.job.JobManager$1@165e96e[Running, pool size = 64,
>> active threads = 64, queued tasks = 5000, completed tasks = 5162]
>>
>> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
>> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
>> at sqlline.SqlLine.print(SqlLine.java:1735)
>> at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
>> at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
>> at sqlline.SqlLine.dispatch(SqlLine.java:821)
>> at sqlline.SqlLine.begin(SqlLine.java:699)
>> at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
>> at sqlline.SqlLine.main(SqlLine.java:424)
>>
>>
>> However, such query like SELECT * FROM SOME_TABLE LIMIT 10 works. Also,
>> other table count query works too. Don't know what causes this kind of
>> issue. Any
>> available hints are appreciated.
>>
>> Thanks,
>> Sun.
>>
>>
>>
>>
>> ________________________________
>> ________________________________
>>
>> CertusNet
>>
>>
>
Re: PhoenixIOException : Task rejected from org.apache.phoenix.job.JobManager
Posted by James Taylor <ja...@apache.org>.
Just to add to what Jan said, if work is being rejected by the thread
pool, you can increase your phoenix.query.queueSize. However, you
don't want it to be so high that the client machine will never be able
to complete the queued tasks. By throwing a rejected exception, the
client is being told "you're giving me too much work to do and I may
never finish it". So there's a balance here.
The other side of the equation is how big you want the work to be
chunked up. This corresponds to how much data will be scanned by each
thread. The bigger the amount of work, the less number of chunks of
work will be queued in the client thread pool. The downside of this is
that you decrease the level of parallelism, thus increasing the
latency you'll see to get query results.
The "chunk size" is controlled through the
phoenix.stats.guidepost.per.region or phoenix.stats.guidepost.width
config properties. You'd set one or the other. Setting
phoenix.stats.guidepost.per.region would give you a "relative" chunk
size depending on how the HBase table is configured. If you set it to
1, then you'd only get a single thread scanning to entire region of a
table. If you set it to 2, then you'd get at most two threads scanning
a region. Setting phoenix.stats.guidepost.width on the other hand
gives you an absolute number of bytes that each scan will perform.
Both these config parameters are server-side settings, so you'd need
to set them in the hbase-sites.xml on each region server, bounce your
cluster, and run UPDATE STATISTICS <table name> to have them take
affect.
Hope this helps. These parameters are documented here:
http://phoenix.apache.org/tuning.html and there's some more
information here: http://phoenix.apache.org/update_statistics.html
Thanks,
James
On Thu, Jan 29, 2015 at 9:01 AM, Jan Fernando <jf...@salesforce.com> wrote:
> Hi Sun,
>
> I believe this means you have exhausted the thread pool and the queue that
> buffers the tasks to service Phoenix queries. Each query will have different
> characteristics in terms of the level of parallelism that Phoenix will use
> to service that query. For example some queries require few (or 1) tasks,
> such as a query with LIMIT 10, other queries that hit many regions and
> guideposts require a higher number of tasks. As you use up the queue and get
> near the max some queries may run if they are enough slots left in the queue
> to service it. So queries requiring a low number of slots like a LIMIT 10
> query might run but others won't.
>
> Hope this helps.
>
> --Jan
>
> On Wed, Jan 28, 2015 at 6:44 PM, sunfl@certusnet.com.cn
> <su...@certusnet.com.cn> wrote:
>>
>> Hi, all
>>
>> I got strange exception hints when doing query like SELECT COUNT(*) FROM
>> SOME_TABLE; through sqlline and the info is
>>
>> java.lang.RuntimeException:
>> org.apache.phoenix.exception.PhoenixIOException: Task
>> org.apache.phoenix.job.JobManager$JobFutureTask@f55dfff rejected from
>>
>> org.apache.phoenix.job.JobManager$1@165e96e[Running, pool size = 64,
>> active threads = 64, queued tasks = 5000, completed tasks = 5162]
>>
>> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
>> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
>> at sqlline.SqlLine.print(SqlLine.java:1735)
>> at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
>> at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
>> at sqlline.SqlLine.dispatch(SqlLine.java:821)
>> at sqlline.SqlLine.begin(SqlLine.java:699)
>> at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
>> at sqlline.SqlLine.main(SqlLine.java:424)
>>
>>
>> However, such query like SELECT * FROM SOME_TABLE LIMIT 10 works. Also,
>> other table count query works too. Don't know what causes this kind of
>> issue. Any
>> available hints are appreciated.
>>
>> Thanks,
>> Sun.
>>
>>
>>
>>
>> ________________________________
>> ________________________________
>>
>> CertusNet
>>
>>
>
Re: PhoenixIOException : Task rejected from org.apache.phoenix.job.JobManager
Posted by Jan Fernando <jf...@salesforce.com>.
Hi Sun,
I believe this means you have exhausted the thread pool and the queue that
buffers the tasks to service Phoenix queries. Each query will have
different characteristics in terms of the level of parallelism that Phoenix
will use to service that query. For example some queries require few (or 1)
tasks, such as a query with LIMIT 10, other queries that hit many regions
and guideposts require a higher number of tasks. As you use up the queue
and get near the max some queries may run if they are enough slots left in
the queue to service it. So queries requiring a low number of slots like a
LIMIT 10 query might run but others won't.
Hope this helps.
--Jan
On Wed, Jan 28, 2015 at 6:44 PM, sunfl@certusnet.com.cn <
sunfl@certusnet.com.cn> wrote:
> Hi, all
>
> I got strange exception hints when doing query like SELECT COUNT(*) FROM
> SOME_TABLE; through sqlline and the info is
>
> java.lang.RuntimeException:
> org.apache.phoenix.exception.PhoenixIOException: Task
> org.apache.phoenix.job.JobManager$JobFutureTask@f55dfff rejected from
>
>
> org.apache.phoenix.job.JobManager$1@165e96e[Running, pool size = 64,
> active threads = 64, queued tasks = 5000, completed tasks = 5162]
>
> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
> at sqlline.SqlLine.print(SqlLine.java:1735)
> at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
> at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
> at sqlline.SqlLine.dispatch(SqlLine.java:821)
> at sqlline.SqlLine.begin(SqlLine.java:699)
> at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
> at sqlline.SqlLine.main(SqlLine.java:424)
>
>
> However, such query like SELECT * FROM SOME_TABLE LIMIT 10 works. Also,
> other table count query works too. Don't know what causes this kind of
> issue. Any
> available hints are appreciated.
>
> Thanks,
> Sun.
>
>
>
>
> ------------------------------
> ------------------------------
>
> CertusNet
>
>
>
回复: PhoenixIOException : Task rejected from org.apache.phoenix.job.JobManager
Posted by "sunfl@certusnet.com.cn" <su...@certusnet.com.cn>.
Hi,
For the exception info, I tried to increase two configs as phoenix.query.threadPoolSize and phoenix.query.queueSize
However, such exception still exists.
Hoping any experts could explain this.
Thanks,
Sun.
CertusNet
发件人： sunfl@certusnet.com.cn
发送时间： 2015-01-29 10:44
收件人： user; dev
主题： PhoenixIOException : Task rejected from org.apache.phoenix.job.JobManager
Hi, all
I got strange exception hints when doing query like SELECT COUNT(*) FROM SOME_TABLE; through sqlline and the info is
java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException: Task org.apache.phoenix.job.JobManager$JobFutureTask@f55dfff rejected from
org.apache.phoenix.job.JobManager$1@165e96e[Running, pool size = 64, active threads = 64, queued tasks = 5000, completed tasks = 5162]
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
at sqlline.SqlLine.print(SqlLine.java:1735)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
at sqlline.SqlLine.dispatch(SqlLine.java:821)
at sqlline.SqlLine.begin(SqlLine.java:699)
at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
at sqlline.SqlLine.main(SqlLine.java:424)
However, such query like SELECT * FROM SOME_TABLE LIMIT 10 works. Also, other table count query works too. Don't know what causes this kind of issue. Any
available hints are appreciated.
Thanks,
Sun.
CertusNet