You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hamado Dene <ha...@yahoo.com.INVALID> on 2021/10/28 14:55:20 UTC
Troubleshooting ipc.RpcServer: (responseTooSlow)
Hi hbase community,
Lately during our nactivities we constantly receive the warn:
2021-10-28 16:45:00,854 WARN [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1635432272849","responsesize":"221799","method":"Scan","param":"scanner_id: 3011016724423115474 number_of_rows: 1000 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: tr \u003cTRUNCATED\u003e","processingtimems":28005,"client":"10.200.86.173:60806","queuetimems":0,"class":"HRegionServer","scandetails":"table: mn1_7491_hinvio region: mn1_7491_hinvio.....}
2021-10-28 16:45:02,527 WARN [RpcServer.default.FPBQ.Fifo.handler=36,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1635432272520","responsesize":"18","method":"Scan","param":"scanner_id: 3011016724423115473 number_of_rows: 1000 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: tr \u003cTRUNCATED\u003e","processingtimems":30006,"client":"10.200.86.130:53476","queuetimems":0,"class":"HRegionServer","scandetails":"table: mn1_7491_hinvio region: mn1_7491_hinvio ..........}
We are still trying to understand which improvements to implement in order to be able to manage the problem.Has anyone ever had the same problem? AndWhat are the configurations on which we can act to have better performance?
Thanks,
Hamado Dene
Re: Troubleshooting ipc.RpcServer: (responseTooSlow)
Posted by Josh Elser <el...@apache.org>.
This is not indicative of an outright problem and can just be a result
of your data and the hardware which you are running HBase on.
Things to note from this data:
1. This RPC will return up to 1000 rows
2. The size of the data returned is not consistent (200KB for one, 18B
for the other)
3. The queuetimems was 0 which means that the full RPC time was spent
scanning data (not waiting to begin)
A general pattern of debugging is to identify one or a few Scans from
your application(s) which are "slow", and diagnose if they are always
slow or sometimes slow. Try to come up with a specific problem statement.
You would have to first make a decision if they _should_ be slow (i.e.
are they reading a significant amount of data and filtering out data?),
or if they are unexpectedly slow.
If they are unexpectedly slow, you would need to start collecting
additional logging from HBase and/or HDFS, and use other JDK-based
performance tracking tools (jvisualvm, flightrecorder, yourkit profiler,
async-profiler) and try to identify if there is a software explanation
for the slowness. Look at the obvious things and keep a list of things
you rule out as "not a problem" vs. "potentially a problem".
Good luck.
On 10/28/21 11:16 AM, Hamado Dene wrote:
> Sorry i forgot to Specify my hbase version:
> Hbase 2.2.6Hadoop: 2.8.5
>
>
>
>
> Il giovedì 28 ottobre 2021, 16:55:40 CEST, Hamado Dene <ha...@yahoo.com.invalid> ha scritto:
>
> Hi hbase community,
> Lately during our nactivities we constantly receive the warn:
>
> 2021-10-28 16:45:00,854 WARN [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1635432272849","responsesize":"221799","method":"Scan","param":"scanner_id: 3011016724423115474 number_of_rows: 1000 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: tr \u003cTRUNCATED\u003e","processingtimems":28005,"client":"10.200.86.173:60806","queuetimems":0,"class":"HRegionServer","scandetails":"table: mn1_7491_hinvio region: mn1_7491_hinvio.....}
>
> 2021-10-28 16:45:02,527 WARN [RpcServer.default.FPBQ.Fifo.handler=36,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1635432272520","responsesize":"18","method":"Scan","param":"scanner_id: 3011016724423115473 number_of_rows: 1000 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: tr \u003cTRUNCATED\u003e","processingtimems":30006,"client":"10.200.86.130:53476","queuetimems":0,"class":"HRegionServer","scandetails":"table: mn1_7491_hinvio region: mn1_7491_hinvio ..........}
> We are still trying to understand which improvements to implement in order to be able to manage the problem.Has anyone ever had the same problem? AndWhat are the configurations on which we can act to have better performance?
> Thanks,
> Hamado Dene
>
Re: Troubleshooting ipc.RpcServer: (responseTooSlow)
Posted by Hamado Dene <ha...@yahoo.com.INVALID>.
Sorry i forgot to Specify my hbase version:
Hbase 2.2.6Hadoop: 2.8.5
Il giovedì 28 ottobre 2021, 16:55:40 CEST, Hamado Dene <ha...@yahoo.com.invalid> ha scritto:
Hi hbase community,
Lately during our nactivities we constantly receive the warn:
2021-10-28 16:45:00,854 WARN [RpcServer.default.FPBQ.Fifo.handler=46,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1635432272849","responsesize":"221799","method":"Scan","param":"scanner_id: 3011016724423115474 number_of_rows: 1000 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: tr \u003cTRUNCATED\u003e","processingtimems":28005,"client":"10.200.86.173:60806","queuetimems":0,"class":"HRegionServer","scandetails":"table: mn1_7491_hinvio region: mn1_7491_hinvio.....}
2021-10-28 16:45:02,527 WARN [RpcServer.default.FPBQ.Fifo.handler=36,queue=1,port=16020] ipc.RpcServer: (responseTooSlow): {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)","starttimems":"1635432272520","responsesize":"18","method":"Scan","param":"scanner_id: 3011016724423115473 number_of_rows: 1000 close_scanner: false next_call_seq: 0 client_handles_partials: true client_handles_heartbeats: tr \u003cTRUNCATED\u003e","processingtimems":30006,"client":"10.200.86.130:53476","queuetimems":0,"class":"HRegionServer","scandetails":"table: mn1_7491_hinvio region: mn1_7491_hinvio ..........}
We are still trying to understand which improvements to implement in order to be able to manage the problem.Has anyone ever had the same problem? AndWhat are the configurations on which we can act to have better performance?
Thanks,
Hamado Dene