You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Abhishek Gupta <ab...@gmail.com> on 2019/05/30 17:04:44 UTC
Does HBase master process need to do listSnapshots
Hi,
We are trying to debug an issue where multiple listSnapshots calls seem to
be running extremely slow RPC calls on HBase Master.
These calls appear to be happening from Master node itself. I wanted to
understand if there is any Master chore service that needs to periodically
do an operation which involves a lisSnapshots?
*{"call":"GetCompletedSnapshots(org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetCompletedSnapshotsRequest)","starttimems":1559233674685,"responsesize":175421,"method":"GetCompletedSnapshots","processingtimems":420615,"client":"10.0.1.193:58296
<http://10.0.1.193:58296>","queuetimems":0,"class":"HMaster"}*
Some of these calls are taking 1000 seconds of processing time.
Thanks in advance,
Abhishek
Re: Does HBase master process need to do listSnapshots
Posted by Abhishek Gupta <ab...@gmail.com>.
Hi Esteban,
Thank you so much for the response, it was indeed the problem. Had a
followup question of this, why is this RPC call not respecting the HBase
RPC time-out
set for 90 seconds by us.
Thanks
On Thu, May 30, 2019 at 10:59 PM Esteban Gutierrez
<es...@cloudera.com.invalid> wrote:
> Hello Abhishek,
>
> The HBase master UI will try to list the snapshots if there are available.
> Since GetCompletedSnapshots goes to the filesystem to scan for all the
> possible snapshots that can take some time depending on the number of
> snapshots or the load of the NN.
>
> thanks,
> esteban.
>
>
> --
> Cloudera, Inc.
>
>
>
> On Thu, May 30, 2019 at 12:05 PM Abhishek Gupta <ab...@gmail.com>
> wrote:
>
> > Hi,
> >
> > We are trying to debug an issue where multiple listSnapshots calls seem
> to
> > be running extremely slow RPC calls on HBase Master.
> > These calls appear to be happening from Master node itself. I wanted to
> > understand if there is any Master chore service that needs to
> periodically
> > do an operation which involves a lisSnapshots?
> >
> >
> *{"call":"GetCompletedSnapshots(org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetCompletedSnapshotsRequest)","starttimems":1559233674685,"responsesize":175421,"method":"GetCompletedSnapshots","processingtimems":420615,"client":"
> > 10.0.1.193:58296
> > <http://10.0.1.193:58296>","queuetimems":0,"class":"HMaster"}*
> > Some of these calls are taking 1000 seconds of processing time.
> >
> > Thanks in advance,
> > Abhishek
> >
>
Re: Does HBase master process need to do listSnapshots
Posted by Abhishek Gupta <ab...@gmail.com>.
Hi Esteban,
Thank you so much for the response, it was indeed the problem. Had a
followup question of this, why is this RPC call not respecting the HBase
RPC time-out
set for 90 seconds by us.
Thanks
On Thu, May 30, 2019 at 10:59 PM Esteban Gutierrez
<es...@cloudera.com.invalid> wrote:
> Hello Abhishek,
>
> The HBase master UI will try to list the snapshots if there are available.
> Since GetCompletedSnapshots goes to the filesystem to scan for all the
> possible snapshots that can take some time depending on the number of
> snapshots or the load of the NN.
>
> thanks,
> esteban.
>
>
> --
> Cloudera, Inc.
>
>
>
> On Thu, May 30, 2019 at 12:05 PM Abhishek Gupta <ab...@gmail.com>
> wrote:
>
> > Hi,
> >
> > We are trying to debug an issue where multiple listSnapshots calls seem
> to
> > be running extremely slow RPC calls on HBase Master.
> > These calls appear to be happening from Master node itself. I wanted to
> > understand if there is any Master chore service that needs to
> periodically
> > do an operation which involves a lisSnapshots?
> >
> >
> *{"call":"GetCompletedSnapshots(org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetCompletedSnapshotsRequest)","starttimems":1559233674685,"responsesize":175421,"method":"GetCompletedSnapshots","processingtimems":420615,"client":"
> > 10.0.1.193:58296
> > <http://10.0.1.193:58296>","queuetimems":0,"class":"HMaster"}*
> > Some of these calls are taking 1000 seconds of processing time.
> >
> > Thanks in advance,
> > Abhishek
> >
>
Re: Does HBase master process need to do listSnapshots
Posted by Esteban Gutierrez <es...@cloudera.com.INVALID>.
Hello Abhishek,
The HBase master UI will try to list the snapshots if there are available.
Since GetCompletedSnapshots goes to the filesystem to scan for all the
possible snapshots that can take some time depending on the number of
snapshots or the load of the NN.
thanks,
esteban.
--
Cloudera, Inc.
On Thu, May 30, 2019 at 12:05 PM Abhishek Gupta <ab...@gmail.com> wrote:
> Hi,
>
> We are trying to debug an issue where multiple listSnapshots calls seem to
> be running extremely slow RPC calls on HBase Master.
> These calls appear to be happening from Master node itself. I wanted to
> understand if there is any Master chore service that needs to periodically
> do an operation which involves a lisSnapshots?
>
> *{"call":"GetCompletedSnapshots(org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetCompletedSnapshotsRequest)","starttimems":1559233674685,"responsesize":175421,"method":"GetCompletedSnapshots","processingtimems":420615,"client":"
> 10.0.1.193:58296
> <http://10.0.1.193:58296>","queuetimems":0,"class":"HMaster"}*
> Some of these calls are taking 1000 seconds of processing time.
>
> Thanks in advance,
> Abhishek
>
Re: Does HBase master process need to do listSnapshots
Posted by Esteban Gutierrez <es...@cloudera.com.INVALID>.
Hello Abhishek,
The HBase master UI will try to list the snapshots if there are available.
Since GetCompletedSnapshots goes to the filesystem to scan for all the
possible snapshots that can take some time depending on the number of
snapshots or the load of the NN.
thanks,
esteban.
--
Cloudera, Inc.
On Thu, May 30, 2019 at 12:05 PM Abhishek Gupta <ab...@gmail.com> wrote:
> Hi,
>
> We are trying to debug an issue where multiple listSnapshots calls seem to
> be running extremely slow RPC calls on HBase Master.
> These calls appear to be happening from Master node itself. I wanted to
> understand if there is any Master chore service that needs to periodically
> do an operation which involves a lisSnapshots?
>
> *{"call":"GetCompletedSnapshots(org.apache.hadoop.hbase.protobuf.generated.MasterProtos$GetCompletedSnapshotsRequest)","starttimems":1559233674685,"responsesize":175421,"method":"GetCompletedSnapshots","processingtimems":420615,"client":"
> 10.0.1.193:58296
> <http://10.0.1.193:58296>","queuetimems":0,"class":"HMaster"}*
> Some of these calls are taking 1000 seconds of processing time.
>
> Thanks in advance,
> Abhishek
>