You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Travis Crawford <tr...@gmail.com> on 2011/03/29 23:29:24 UTC
Identify ephemeral node owner
Hey zookeepers -
Is there a way to map the ephemeral node owner session back to the
process holding that node open? For example, given the following znode
stat output from the python client I'm interested in finding who has
this open.
{'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
Grep'ing through zookeeper.log I found lines like follows referencing
the ephemeralOwner number, however, nothing useful in tracking down
the process holding it open.
2011-03-29 13:12:50,083 - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
client: 301083794498035291
Thanks!
Travis
Re: Identify ephemeral node owner
Posted by Travis Crawford <tr...@gmail.com>.
Thanks for the info! That initial session creation log line would be
quite handy if it hadn't been rotated away in this case :) I'll
probably change the registration code to include info about who
created the node - in this case I'll track the process down via some
other route.
ZOOKEEPER-829 is interesting btw, sort of like /proc - nice!
--travis
On Tue, Mar 29, 2011 at 2:47 PM, Patrick Hunt <ph...@apache.org> wrote:
> Hi Travis,
>
> as Mahadev mentioned, when a client establishes a session with a
> server the host id is logged to the server log:
> 2011-03-29 14:41:59,741 - INFO [SyncThread:0:NIOServerCnxn@1580] -
> Established session 0x12f0390e7520000 with negotiated timeout 30000
> for client /127.0.0.1:47789
>
> You could examine the ephemeral owner and grep the logs for this
> (that's what I typically do).
>
> A second more complex option; the client could note it's session id,
> along with some other identifying information
> (host/process/user/etc...) as a ZNode as part of it's joining the
> cluster. This would obv require changes on the client side though.
>
> There's also this jira, might be a good idea for us to capture some
> client specific information here (although we'd have to watch out for
> security issues with this, exposing this information only to
> appropriate parties...)
> https://issues.apache.org/jira/browse/ZOOKEEPER-829
>
> Patrick
>
> On Tue, Mar 29, 2011 at 2:38 PM, Mahadev Konar <ma...@apache.org> wrote:
>> Travis,
>> the only way to track an ephemeral node is the session id, in your case:
>>
>> 301083794498035291. You can find which ip has this session by mining
>> the logs for zookeeper servers, but for pin pointing which process
>> unfortunately you'll have to look at your process logs and see which
>> one created that session.
>>
>> thanks
>> mahadev
>>
>> The session id is the session which the znode belongs to. For finding
>> the process that m
>>
>> On Tue, Mar 29, 2011 at 2:29 PM, Travis Crawford
>> <tr...@gmail.com> wrote:
>>> Hey zookeepers -
>>>
>>> Is there a way to map the ephemeral node owner session back to the
>>> process holding that node open? For example, given the following znode
>>> stat output from the python client I'm interested in finding who has
>>> this open.
>>>
>>> {'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
>>> 'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
>>> 301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
>>> 1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
>>>
>>>
>>> Grep'ing through zookeeper.log I found lines like follows referencing
>>> the ephemeralOwner number, however, nothing useful in tracking down
>>> the process holding it open.
>>>
>>> 2011-03-29 13:12:50,083 - INFO
>>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
>>> client: 301083794498035291
>>>
>>>
>>> Thanks!
>>> Travis
>>>
>>
>>
>>
>> --
>> thanks
>> mahadev
>> @mahadevkonar
>>
>
Re: Identify ephemeral node owner
Posted by Ted Dunning <te...@gmail.com>.
This sort of thing is exactly the rationale for extensible schemas such as
you get from JSON, Avro or protobufs.
On Wed, Mar 30, 2011 at 4:28 PM, Travis Crawford
<tr...@gmail.com>wrote:
> On Tue, Mar 29, 2011 at 3:03 PM, Ted Dunning <te...@gmail.com>
> wrote:
> > The simplest (though probably least acceptable in most cases) solution is
> to
> > put the process host and pid into the ephemeral file in some form.
> >
>
> To close the loop... I ended up doing exactly this. The app already
> ...
Thanks for the pointers, y'all! Not a general solution but works well
> in this case.
Another way to look at this is to say that the solution is incredibly
general (you can have your bytes, your way).
Re: Identify ephemeral node owner
Posted by Travis Crawford <tr...@gmail.com>.
On Tue, Mar 29, 2011 at 3:03 PM, Ted Dunning <te...@gmail.com> wrote:
> The simplest (though probably least acceptable in most cases) solution is to
> put the process host and pid into the ephemeral file in some form.
>
To close the loop... I ended up doing exactly this. The app already
registered itself with:
https://github.com/twitter/commons/blob/master/src/java/com/twitter/common/zookeeper/Group.java#L215
So I wrote a Supplier that puts the hostname + pid into the znode as
json, which lets us add more stuff over time if needed.
public static JsonSupplier newProcessDescriptionSupplier() {
RuntimeMXBean bean = ManagementFactory.getRuntimeMXBean();
String[] parts = bean.getName().split("@");
JSONObject j = new JSONObject();
try {
j.put("pid", parts[0]);
j.put("hostname", parts[1]);
} catch (JSONException e) {
LOG.error("Unable to populate supplier with initial values!", e);
}
return new JsonSupplier(j);
}
Thanks for the pointers, y'all! Not a general solution but works well
in this case.
--travis
> On Tue, Mar 29, 2011 at 2:47 PM, Patrick Hunt <ph...@apache.org> wrote:
>>
>> Hi Travis,
>>
>> as Mahadev mentioned, when a client establishes a session with a
>> server the host id is logged to the server log:
>> 2011-03-29 14:41:59,741 - INFO [SyncThread:0:NIOServerCnxn@1580] -
>> Established session 0x12f0390e7520000 with negotiated timeout 30000
>> for client /127.0.0.1:47789
>>
>> You could examine the ephemeral owner and grep the logs for this
>> (that's what I typically do).
>>
>> A second more complex option; the client could note it's session id,
>> along with some other identifying information
>> (host/process/user/etc...) as a ZNode as part of it's joining the
>> cluster. This would obv require changes on the client side though.
>>
>> There's also this jira, might be a good idea for us to capture some
>> client specific information here (although we'd have to watch out for
>> security issues with this, exposing this information only to
>> appropriate parties...)
>> https://issues.apache.org/jira/browse/ZOOKEEPER-829
>>
>> Patrick
>>
>> On Tue, Mar 29, 2011 at 2:38 PM, Mahadev Konar <ma...@apache.org> wrote:
>> > Travis,
>> > the only way to track an ephemeral node is the session id, in your
>> > case:
>> >
>> > 301083794498035291. You can find which ip has this session by mining
>> > the logs for zookeeper servers, but for pin pointing which process
>> > unfortunately you'll have to look at your process logs and see which
>> > one created that session.
>> >
>> > thanks
>> > mahadev
>> >
>> > The session id is the session which the znode belongs to. For finding
>> > the process that m
>> >
>> > On Tue, Mar 29, 2011 at 2:29 PM, Travis Crawford
>> > <tr...@gmail.com> wrote:
>> >> Hey zookeepers -
>> >>
>> >> Is there a way to map the ephemeral node owner session back to the
>> >> process holding that node open? For example, given the following znode
>> >> stat output from the python client I'm interested in finding who has
>> >> this open.
>> >>
>> >> {'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
>> >> 'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
>> >> 301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
>> >> 1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
>> >>
>> >>
>> >> Grep'ing through zookeeper.log I found lines like follows referencing
>> >> the ephemeralOwner number, however, nothing useful in tracking down
>> >> the process holding it open.
>> >>
>> >> 2011-03-29 13:12:50,083 - INFO
>> >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
>> >> client: 301083794498035291
>> >>
>> >>
>> >> Thanks!
>> >> Travis
>> >>
>> >
>> >
>> >
>> > --
>> > thanks
>> > mahadev
>> > @mahadevkonar
>> >
>
>
Re: Identify ephemeral node owner
Posted by Ted Dunning <te...@gmail.com>.
The simplest (though probably least acceptable in most cases) solution is to
put the process host and pid into the ephemeral file in some form.
On Tue, Mar 29, 2011 at 2:47 PM, Patrick Hunt <ph...@apache.org> wrote:
> Hi Travis,
>
> as Mahadev mentioned, when a client establishes a session with a
> server the host id is logged to the server log:
> 2011-03-29 14:41:59,741 - INFO [SyncThread:0:NIOServerCnxn@1580] -
> Established session 0x12f0390e7520000 with negotiated timeout 30000
> for client /127.0.0.1:47789
>
> You could examine the ephemeral owner and grep the logs for this
> (that's what I typically do).
>
> A second more complex option; the client could note it's session id,
> along with some other identifying information
> (host/process/user/etc...) as a ZNode as part of it's joining the
> cluster. This would obv require changes on the client side though.
>
> There's also this jira, might be a good idea for us to capture some
> client specific information here (although we'd have to watch out for
> security issues with this, exposing this information only to
> appropriate parties...)
> https://issues.apache.org/jira/browse/ZOOKEEPER-829
>
> Patrick
>
> On Tue, Mar 29, 2011 at 2:38 PM, Mahadev Konar <ma...@apache.org> wrote:
> > Travis,
> > the only way to track an ephemeral node is the session id, in your case:
> >
> > 301083794498035291. You can find which ip has this session by mining
> > the logs for zookeeper servers, but for pin pointing which process
> > unfortunately you'll have to look at your process logs and see which
> > one created that session.
> >
> > thanks
> > mahadev
> >
> > The session id is the session which the znode belongs to. For finding
> > the process that m
> >
> > On Tue, Mar 29, 2011 at 2:29 PM, Travis Crawford
> > <tr...@gmail.com> wrote:
> >> Hey zookeepers -
> >>
> >> Is there a way to map the ephemeral node owner session back to the
> >> process holding that node open? For example, given the following znode
> >> stat output from the python client I'm interested in finding who has
> >> this open.
> >>
> >> {'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
> >> 'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
> >> 301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
> >> 1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
> >>
> >>
> >> Grep'ing through zookeeper.log I found lines like follows referencing
> >> the ephemeralOwner number, however, nothing useful in tracking down
> >> the process holding it open.
> >>
> >> 2011-03-29 13:12:50,083 - INFO
> >> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
> >> client: 301083794498035291
> >>
> >>
> >> Thanks!
> >> Travis
> >>
> >
> >
> >
> > --
> > thanks
> > mahadev
> > @mahadevkonar
> >
>
Re: Identify ephemeral node owner
Posted by Patrick Hunt <ph...@apache.org>.
Hi Travis,
as Mahadev mentioned, when a client establishes a session with a
server the host id is logged to the server log:
2011-03-29 14:41:59,741 - INFO [SyncThread:0:NIOServerCnxn@1580] -
Established session 0x12f0390e7520000 with negotiated timeout 30000
for client /127.0.0.1:47789
You could examine the ephemeral owner and grep the logs for this
(that's what I typically do).
A second more complex option; the client could note it's session id,
along with some other identifying information
(host/process/user/etc...) as a ZNode as part of it's joining the
cluster. This would obv require changes on the client side though.
There's also this jira, might be a good idea for us to capture some
client specific information here (although we'd have to watch out for
security issues with this, exposing this information only to
appropriate parties...)
https://issues.apache.org/jira/browse/ZOOKEEPER-829
Patrick
On Tue, Mar 29, 2011 at 2:38 PM, Mahadev Konar <ma...@apache.org> wrote:
> Travis,
> the only way to track an ephemeral node is the session id, in your case:
>
> 301083794498035291. You can find which ip has this session by mining
> the logs for zookeeper servers, but for pin pointing which process
> unfortunately you'll have to look at your process logs and see which
> one created that session.
>
> thanks
> mahadev
>
> The session id is the session which the znode belongs to. For finding
> the process that m
>
> On Tue, Mar 29, 2011 at 2:29 PM, Travis Crawford
> <tr...@gmail.com> wrote:
>> Hey zookeepers -
>>
>> Is there a way to map the ephemeral node owner session back to the
>> process holding that node open? For example, given the following znode
>> stat output from the python client I'm interested in finding who has
>> this open.
>>
>> {'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
>> 'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
>> 301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
>> 1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
>>
>>
>> Grep'ing through zookeeper.log I found lines like follows referencing
>> the ephemeralOwner number, however, nothing useful in tracking down
>> the process holding it open.
>>
>> 2011-03-29 13:12:50,083 - INFO
>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
>> client: 301083794498035291
>>
>>
>> Thanks!
>> Travis
>>
>
>
>
> --
> thanks
> mahadev
> @mahadevkonar
>
Re: Identify ephemeral node owner
Posted by Mahadev Konar <ma...@apache.org>.
Travis,
the only way to track an ephemeral node is the session id, in your case:
301083794498035291. You can find which ip has this session by mining
the logs for zookeeper servers, but for pin pointing which process
unfortunately you'll have to look at your process logs and see which
one created that session.
thanks
mahadev
The session id is the session which the znode belongs to. For finding
the process that m
On Tue, Mar 29, 2011 at 2:29 PM, Travis Crawford
<tr...@gmail.com> wrote:
> Hey zookeepers -
>
> Is there a way to map the ephemeral node owner session back to the
> process holding that node open? For example, given the following znode
> stat output from the python client I'm interested in finding who has
> this open.
>
> {'pzxid': 17302254734L, 'ctime': 1301279980630L, 'aversion': 0,
> 'mzxid': 17302254734L, 'numChildren': 0, 'ephemeralOwner':
> 301083794498035291L, 'version': 0, 'dataLength': 0, 'mtime':
> 1301279980630L, 'cversion': 0, 'czxid': 17302254734L}
>
>
> Grep'ing through zookeeper.log I found lines like follows referencing
> the ephemeralOwner number, however, nothing useful in tracking down
> the process holding it open.
>
> 2011-03-29 13:12:50,083 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@95] - Revalidating
> client: 301083794498035291
>
>
> Thanks!
> Travis
>
--
thanks
mahadev
@mahadevkonar