You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Konstantinos Kougios <ko...@googlemail.com> on 2015/09/28 16:41:25 UTC
sqlline.py, timeout after 60secs, is there a way to increase the
timeout period?
I have a fairly big table on a noy-so-fairly-powerfull cluster, so it
takes a lot of time for queries to respond. I don't mind that but it
times out for many queries :
0: jdbc:phoenix:nn.lan> select count(*) from words;
+------------------------------------------+
| COUNT(1) |
+------------------------------------------+
java.lang.RuntimeException:
org.apache.phoenix.exception.PhoenixIOException:
org.apache.phoenix.exception.PhoenixIOException: Failed after
attempts=1, exceptions:
Mon Sep 28 15:32:25 BST 2015,
RpcRetryingCaller{globalStartTime=1443450685716, pause=100, retries=1},
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Mon Sep 28 15:32:25 BST 2015,
RpcRetryingCaller{globalStartTime=1443450685716, pause=100, retries=1},
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Mon Sep 28 15:32:25 BST 2015,
RpcRetryingCaller{globalStartTime=1443450685716, pause=100, retries=1},
java.io.IOException: Call to d2.lan/192.168.0.30:16020 failed on local
exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
id=842, waitTime=60001, operationTimeout=60000 expired.
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
at sqlline.SqlLine.print(SqlLine.java:1653)
at sqlline.Commands.execute(Commands.java:833)
at sqlline.Commands.sql(Commands.java:732)
at sqlline.SqlLine.dispatch(SqlLine.java:808)
at sqlline.SqlLine.begin(SqlLine.java:681)
at sqlline.SqlLine.start(SqlLine.java:398)
at sqlline.SqlLine.main(SqlLine.java:292)
0: jdbc:phoenix:nn.lan> Closing: org.apache.phoenix.jdbc.PhoenixConnection
I did try some settings on hbase-site without luck:
phoenix.query.timeoutMs =6000000 hbase.client.operation.timeout =1200000 hbase.client.backpressure.enabled =true hbase.client.retries.number =1
Any ideas how can this be fixed?
It seems the problem is that the timeout for iterating through results
is 60secs. I assume if it doesn't get 1 result within that period, it
times out. Since this is a count(*) query with only 1 row, it does timeout.
Thanks
Re: sqlline.py, timeout after 60secs, is there a way to increase the
timeout period?
Posted by Konstantinos Kougios <ko...@googlemail.com>.
hmm, my 2nd problem is probably caused by region servers running out of
memory. 2 out of 3 region servers reached the mem limit of 4GB
So my settings are -Xmx4G and
phoenix.query.maxServerCacheBytes = 512000000
phoenix.query.maxGlobalMemoryPercentage = 50
I'll try to put maxGlobalMemoryPercentage to 25% but I don't think it
will help. Doesn't phoenix cap the memory usage on heavy queries? What
is using the memory?
On 28/09/15 15:49, Lukáš Lalinský wrote:
> You need to set "hbase.rpc.timeout" to the same value as you have for
> "phoenix.query.timeoutMs".
>
> It seems that in the pre-Apache version of Phoenix it was set
> automatically:
>
> https://issues.apache.org/jira/browse/PHOENIX-269?focusedCommentId=14681924&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14681924
>
> Lukas
>
>
> On Mon, Sep 28, 2015 at 4:41 PM, Konstantinos Kougios
> <kostas.kougios@googlemail.com <ma...@googlemail.com>>
> wrote:
>
> I have a fairly big table on a noy-so-fairly-powerfull cluster, so
> it takes a lot of time for queries to respond. I don't mind that
> but it times out for many queries :
>
> 0: jdbc:phoenix:nn.lan> select count(*) from words;
> +------------------------------------------+
> | COUNT(1) |
> +------------------------------------------+
> java.lang.RuntimeException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=1, exceptions:
> Mon Sep 28 15:32:25 BST 2015,
> RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
> retries=1},
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=1, exceptions:
> Mon Sep 28 15:32:25 BST 2015,
> RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
> retries=1},
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=1, exceptions:
> Mon Sep 28 15:32:25 BST 2015,
> RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
> retries=1}, java.io.IOException: Call to d2.lan/192.168.0.30:16020
> <http://192.168.0.30:16020> failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=842,
> waitTime=60001, operationTimeout=60000 expired.
>
>
>
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
> at sqlline.SqlLine.print(SqlLine.java:1653)
> at sqlline.Commands.execute(Commands.java:833)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> 0: jdbc:phoenix:nn.lan> Closing:
> org.apache.phoenix.jdbc.PhoenixConnection
>
>
> I did try some settings on hbase-site without luck:
>
>
> phoenix.query.timeoutMs =6000000 hbase.client.operation.timeout =1200000 hbase.client.backpressure.enabled =true hbase.client.retries.number =1
>
> Any ideas how can this be fixed?
>
> It seems the problem is that the timeout for iterating through
> results is 60secs. I assume if it doesn't get 1 result within that
> period, it times out. Since this is a count(*) query with only 1
> row, it does timeout.
>
> Thanks
>
>
Re: sqlline.py, timeout after 60secs, is there a way to increase the
timeout period?
Posted by Konstantinos Kougios <ko...@googlemail.com>.
Thanks Lukas, that seems to have fixed it. I've set it both on my
cluster and locally to my local hbase installation for it to work.
Now I am getting a different problem when doing a query:
0: jdbc:phoenix:nn.lan> select count(word),word from words group by word
order by count(word) desc limit 10;
+------------------------------------------+------------------------------------------+
| COUNT(WORD) | WORD |
+------------------------------------------+------------------------------------------+
15/09/28 17:25:51 WARN client.ScannerCallable: Ignore, probably already
closed
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: Name: 103, already closed?
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:322)
at
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:357)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:195)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:142)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
at
org.apache.hadoop.hbase.client.StatsTrackingRpcRetryingCaller.callWithoutRetries(StatsTrackingRpcRetryingCaller.java:56)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:258)
at
org.apache.hadoop.hbase.client.ClientScanner.possiblyNextScanner(ClientScanner.java:241)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:532)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:364)
at
org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:55)
at
org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:107)
at
org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:240)
at
org.apache.phoenix.iterate.OrderedResultIterator.peek(OrderedResultIterator.java:263)
at
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:112)
at
org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException):
org.apache.hadoop.hbase.UnknownScannerException: Name: 103, already closed?
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2223)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1196)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)
at
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:355)
... 21 more
Do you know if it is related to phoenix.query.keepAliveMs ? (currently
my cluster is busy but will give it a try later on)
On 28/09/15 15:49, Lukáš Lalinský wrote:
> You need to set "hbase.rpc.timeout" to the same value as you have for
> "phoenix.query.timeoutMs".
>
> It seems that in the pre-Apache version of Phoenix it was set
> automatically:
>
> https://issues.apache.org/jira/browse/PHOENIX-269?focusedCommentId=14681924&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14681924
>
> Lukas
>
>
> On Mon, Sep 28, 2015 at 4:41 PM, Konstantinos Kougios
> <kostas.kougios@googlemail.com <ma...@googlemail.com>>
> wrote:
>
> I have a fairly big table on a noy-so-fairly-powerfull cluster, so
> it takes a lot of time for queries to respond. I don't mind that
> but it times out for many queries :
>
> 0: jdbc:phoenix:nn.lan> select count(*) from words;
> +------------------------------------------+
> | COUNT(1) |
> +------------------------------------------+
> java.lang.RuntimeException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=1, exceptions:
> Mon Sep 28 15:32:25 BST 2015,
> RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
> retries=1},
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=1, exceptions:
> Mon Sep 28 15:32:25 BST 2015,
> RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
> retries=1},
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=1, exceptions:
> Mon Sep 28 15:32:25 BST 2015,
> RpcRetryingCaller{globalStartTime=1443450685716, pause=100,
> retries=1}, java.io.IOException: Call to d2.lan/192.168.0.30:16020
> <http://192.168.0.30:16020> failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=842,
> waitTime=60001, operationTimeout=60000 expired.
>
>
>
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
> at sqlline.SqlLine.print(SqlLine.java:1653)
> at sqlline.Commands.execute(Commands.java:833)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> 0: jdbc:phoenix:nn.lan> Closing:
> org.apache.phoenix.jdbc.PhoenixConnection
>
>
> I did try some settings on hbase-site without luck:
>
>
> phoenix.query.timeoutMs =6000000 hbase.client.operation.timeout =1200000 hbase.client.backpressure.enabled =true hbase.client.retries.number =1
>
> Any ideas how can this be fixed?
>
> It seems the problem is that the timeout for iterating through
> results is 60secs. I assume if it doesn't get 1 result within that
> period, it times out. Since this is a count(*) query with only 1
> row, it does timeout.
>
> Thanks
>
>
Re: sqlline.py, timeout after 60secs, is there a way to increase the
timeout period?
Posted by Lukáš Lalinský <la...@gmail.com>.
You need to set "hbase.rpc.timeout" to the same value as you have for
"phoenix.query.timeoutMs".
It seems that in the pre-Apache version of Phoenix it was set automatically:
https://issues.apache.org/jira/browse/PHOENIX-269?focusedCommentId=14681924&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14681924
Lukas
On Mon, Sep 28, 2015 at 4:41 PM, Konstantinos Kougios <
kostas.kougios@googlemail.com> wrote:
> I have a fairly big table on a noy-so-fairly-powerfull cluster, so it
> takes a lot of time for queries to respond. I don't mind that but it times
> out for many queries :
>
> 0: jdbc:phoenix:nn.lan> select count(*) from words;
> +------------------------------------------+
> | COUNT(1) |
> +------------------------------------------+
> java.lang.RuntimeException:
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=1,
> exceptions:
> Mon Sep 28 15:32:25 BST 2015,
> RpcRetryingCaller{globalStartTime=1443450685716, pause=100, retries=1},
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=1, exceptions:
> Mon Sep 28 15:32:25 BST 2015,
> RpcRetryingCaller{globalStartTime=1443450685716, pause=100, retries=1},
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=1, exceptions:
> Mon Sep 28 15:32:25 BST 2015,
> RpcRetryingCaller{globalStartTime=1443450685716, pause=100, retries=1},
> java.io.IOException: Call to d2.lan/192.168.0.30:16020 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=842,
> waitTime=60001, operationTimeout=60000 expired.
>
>
>
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
> at sqlline.SqlLine.print(SqlLine.java:1653)
> at sqlline.Commands.execute(Commands.java:833)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> 0: jdbc:phoenix:nn.lan> Closing: org.apache.phoenix.jdbc.PhoenixConnection
>
>
> I did try some settings on hbase-site without luck:
>
>
> phoenix.query.timeoutMs = 6000000hbase.client.operation.timeout = 1200000hbase.client.backpressure.enabled = truehbase.client.retries.number = 1
>
> Any ideas how can this be fixed?
>
> It seems the problem is that the timeout for iterating through results is
> 60secs. I assume if it doesn't get 1 result within that period, it times
> out. Since this is a count(*) query with only 1 row, it does timeout.
>
> Thanks
>