You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2025/09/25 00:15:58 UTC

[I] `ZkBaseDataAccessor.set` should either success or throw exception, but not both [helix]

Jackie-Jiang opened a new issue, #3072:
URL: https://github.com/apache/helix/issues/3072

   Continuation on #2685 
   
   When setting a record in ZK, we need deterministic behavior, were the call should either success or fail (throw exception). Currently the call might change the record but also throw exception. Do you see a workaround to get deterministic result?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [I] `ZkBaseDataAccessor.set` should either success or throw exception, but not both [helix]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #3072:
URL: https://github.com/apache/helix/issues/3072#issuecomment-3930111897

   Do you mean creating a separate `ZkClient` and manage it ourself instead of using the one from `ZKHelixManager`?
   IMO this needs to be handled within Helix because Helix also relies on the response of `ZkBaseDataAccessor` to determine the next step, and requires deterministic behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [I] `ZkBaseDataAccessor.set` should either success or throw exception, but not both [helix]

Posted by "junkaixue (via GitHub) via reviews" <re...@helix.apache.org>.
junkaixue commented on issue #3072:
URL: https://github.com/apache/helix/issues/3072#issuecomment-6672621972

   Commit in. Close this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [I] `ZkBaseDataAccessor.set` should either success or throw exception, but not both [helix]

Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue commented on issue #3072:
URL: https://github.com/apache/helix/issues/3072#issuecomment-3924863810

   If you see ZKInterruptException, what is foundamtentally somewhere call the zkclient.disconnect(). The right solution is try to create non-emphereal zkclient as a static client disconnect along with the application lifecycle.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [I] `ZkBaseDataAccessor.set` should either success or throw exception, but not both [helix]

Posted by "junkaixue (via GitHub) via reviews" <re...@helix.apache.org>.
junkaixue closed issue #3072: `ZkBaseDataAccessor.set` should either success or throw exception, but not both
URL: https://github.com/apache/helix/issues/3072


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [I] `ZkBaseDataAccessor.set` should either success or throw exception, but not both [helix]

Posted by "junkaixue (via GitHub) via reviews" <re...@helix.apache.org>.
junkaixue commented on issue #3072:
URL: https://github.com/apache/helix/issues/3072#issuecomment-6671400097

   Thanks for the follow-up, and apologies for the slow response. I traced this through both our ZkClient and the ZooKeeper client it wraps, and want to lay out what is happening, what we will fix, and a workaround you can use today.
   
   Where the ambiguity comes from
   
   The InterruptedException in the #2685 trace is thrown from ClientCnxn.submitRequest inside the ZooKeeper client library. That method queues the request packet to the send thread first and only then blocks in packet.wait() for the reply. If the calling thread is interrupted while waiting, the exception propagates up through our ZkClient and ZkBaseDataAccessor, but the packet is already in the outgoing queue and the send thread delivers it to the server regardless. So the server may apply the write while the caller sees an exception.
   
   One correction to my earlier comment: ZkClient.close() does not interrupt caller threads. It only interrupts its own event thread and async retry thread. A caller blocked in a write when the client closes gets CONNECTIONLOSS, not InterruptedException. The interrupt in your trace hit jersey-server-managed-async-executor-24, which means the Jersey executor was being shut down concurrently with disconnect(). A separately managed ZkClient would not have helped, since the interrupt lands on the calling thread regardless of which client it is using.
   
   What Helix will fix
   
   There are two layers in Helix and we should fix both.
   
   1. ZkBaseDataAccessor.doSet (and doCreate, doUpdate, doRemove) catches Exception broadly and turns ZkInterruptedException into RetCode.ERROR, so set() returns false. That reports "not applied" when the true state is "unknown". We will rethrow ZkInterruptedException the same way ZkBadVersionException is already rethrown. The contract becomes: true means applied, false means definitely not applied, exception means the operation was aborted and the outcome is unknown.
   2. In ZkClient itself, we can narrow the unknown window considerably. ZooKeeper guarantees that every queued packet eventually finishes, either with the server's reply or with CONNECTIONLOSS / SESSIONEXPIRED from the send thread's cleanup. So when the calling thread is interrupted mid-write, ZkClient can issue the write through the async API and, on interrupt, keep waiting for that already-queued packet's completion for a bounded time before giving up, then restore the interrupt flag. In the common case where the connection is healthy, the reply arrives in milliseconds and the caller gets a definitive result. The residual unknown case shrinks to "interrupted and the connection is also broken", which is as deterministic as the ZooKeeper protocol allows. I'll size this one separately since it touches the retry path and we need to keep interrupt-to-abort working for the disconnect hang we fixed recently.
   
   Workaround available now
   
   ZkInterruptedException's constructor re-sets the thread's interrupt flag before the accessor swallows it. So after set() returns false, you can check Thread.currentThread().isInterrupted(). If it is set, treat the result as unknown and re-read the IdealState rather than assuming the write failed. That distinguishes the ambiguous case from a real failure on current releases.
   
   I'll link the PRs here once they are up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org