You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@curator.apache.org by "jira-importer (via GitHub)" <gi...@apache.org> on 2025/01/21 13:12:25 UTC

[I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

jira-importer opened a new issue, #578:
URL: https://github.com/apache/curator/issues/578

   <p>I am attaching a simple test to demonstrate the problem:</p>
   
   <p>Single threaded 'for' loop that creates children 0 to N, the events are not received in order.</p>
   
   ---
   <details><summary><i>Originally reported by <a title="Benjamin Jaton" href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=benjamin.jaton">benjamin.jaton</a>, imported from: <a href="https://issues.apache.org/jira/browse/CURATOR-57" target="_blank">Event order lost with PathChildrenCache</a></i></summary>
   <i><ul>
   <li><b>status</b>: Closed
   <li><b>priority</b>: Major
   <li><b>resolution</b>: Invalid
   <li><b>resolved</b>: 2013-09-26T16:06:29+01:00
   <li><b>imported</b>: 2025-01-21
   </ul></i>
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699573

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=benjamin.jaton">benjamin.jaton</a>:</i>
   <p>So we do have the Stat object right? The data is optional but the Stat is always fetched.</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699556

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
   <p>Those stat objects are retrieved one at a time. See internalRebuildNode(). So, there's still no way that PathChildrenCache can return events in order. </p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699590

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
   <p>Sorry - the above comes across as terse. I really don't know how to solve this problem. If you know of a way, please describe it.</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699551

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=benjamin.jaton">benjamin.jaton</a>:</i>
   <p>I believe the Stat object is already available in the event through event.getData().getStat():</p>
   
   <p>@Override<br/>
   public void childEvent(CuratorFramework curator, PathChildrenCacheEvent event) throws Exception {<br/>
     System.out.println("CZXID="+ event.getData().getStat().getCzxid());<br/>
     // more stuff<br/>
   }</p>
   
   <p>Am I missing something?</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699583

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
   <p><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=benjamin.jaton" class="user-hover" rel="benjamin.jaton">Benjamin Jaton</a> - either I don't understand you or you don't understand me. You want to get creation events in order. I've told you that each node's stat object is independently retrieved. I don't see how these two facts can be reconciled. If you do, please provide an implementation.</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699563

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=benjamin.jaton">benjamin.jaton</a>:</i>
   <p>The test I attached previously doesn't invoke internalRebuildNode(), even when I use event.getData().getStat().<br/>
   Instead it fetches the Stat object in processChildren() (see getDataAndStat(fullPath); ).</p>
   
   <p>If you have a look at this test, you will see that I am able to access the Stat object.</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699599

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=benjamin.jaton">benjamin.jaton</a>:</i>
   <p>Well I think there is/has been some confusion about internalRebuildNode().</p>
   
   <p>It is up to the PathChildrenCache implementation to retrieve the Stat object independently or not. We could imagine another implementation that would have preferred to iterate over the result of the getChildren() watcher, wait for all the corresponding Stat objects to be gathered before making an informed decision on which events should be dispatched first.</p>
   
   <p>I agree that this could be achieved in another recipe, based on PathChildrenCache but that would be single threaded (makes no sense to dispatch the events in parallel otherwise we could lose the ordering).</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699608

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
   <p>PathChildrenCache is not intended to preserve message ordering. A separate recipe should be written to accomplish this.</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699547

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
   <p>The problem there is that you have to make a call on each individual node to get the Stat object. So, there's no way to do this "inline". All the details have to be fetched and then you can construct the ordering. This suggests a higher-level recipe that uses PathChildrenCache internally. It could wait until node changes settle or a pre-defined timing threshold and then construct a new message with the ordering that you need.</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699567

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
   <p>But PathChildrenCache calls internalRebuildNode(). The algorithm (loosely) is call getChildren() with a watch. For each node returned by getChildren, call internalRebuildNode() to get stat and (optionally) the data. When the watcher fires, do it all again.</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699544

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=benjamin.jaton">benjamin.jaton</a>:</i>
   <p>I think we could use ZooKeeper's transaction IDs:<br/>
   "Every change to the ZooKeeper state receives a stamp in the form of a zxid (ZooKeeper Transaction Id). This exposes the total ordering of all changes to ZooKeeper. Each change will have a unique zxid and if zxid1 is smaller than zxid2 then zxid1 happened before zxid2."</p>
   
   <p>In the case of an event, we could use the czxid in the Stat object to have the order of creation of the children:<br/>
   "czxid: The zxid of the change that caused this znode to be created."</p>
   
   <p>Source:<br/>
   <a href="http://zookeeper.apache.org/doc/r3.4.5/zookeeperProgrammers.html#sc_zkStatStructure" class="external-link" target="_blank" rel="nofollow noopener">http://zookeeper.apache.org/doc/r3.4.5/zookeeperProgrammers.html#sc_zkStatStructure</a></p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699603

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
   <p>PathChildrenCache works this way already. The issue is how you determine when to send messages? In a dynamic ZK ensemble nodes are being changed all the time. You could send messages after each getChildren() batch is processed but this is arbitrary. Also, in order to truly ensure ordering you'd likely need to re-query ZooKeeper inside of the Watchers which is an anti-pattern (see <a href="https://cwiki.apache.org/confluence/display/CURATOR/TN1" class="external-link" target="_blank" rel="nofollow noopener">https://cwiki.apache.org/confluence/display/CURATOR/TN1</a>).</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699535

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
   <p>I don't see how this can be supported. A watcher is set for the parent node by calling getChildren(). When the watcher triggers, getChildren() is called. getChildren() returns a list of name - no ordering is provided. Unless a method of solving this can be suggested, I'll close this issue.</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [CURATOR-57] Event order lost with PathChildrenCache [curator]

Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #578:
URL: https://github.com/apache/curator/issues/578#issuecomment-2604699532

   <i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=benjamin.jaton">benjamin.jaton</a>:</i>
   <p>Simple test to demonstrate the issue.</p>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org