[GitHub] [hudi] guanziyue commented on pull request #4264: [HUDI-2875] Make HoodieParquetWriter Thread safe and memory executor …

[GitBox <gi...@apache.org>]

guanziyue commented on pull request #4264:
URL: https://github.com/apache/hudi/pull/4264#issuecomment-1064821297


   > @guanziyue thank you for taking the time to troubleshoot this concurrency issues and implement the fix!
   > 
   > I echo @vinothchandar concerns and i think we're taking a step a bit too far -- `ParquetWriter` is not assumed to be thread-safe, neither do i believe we should make it such.
   > 
   > Instead, i believe we should just resolve the problem with its concurrent access (which you already did) and make sure we make it clear that `ParquetWriter` is not thread-safe so its usage need to be properly guarded externally.
   
   Hi @alexeykudinkin, may I know if your concern is "adding a lock to parquetWriter" or "adding a lock to hot path"? I'm afraid that it is difficult to come up with a method to guarantee this problem is totally solved except adding a signal to hot path. Producer need to check if current thread is interrupted and response to it in a reasonable time or consumer need to immediately reject any writing just after close method is called, which also need a lock on hot path. For producer solution, we can have a lock-free check. For consumer, we may use volatile rather than a lock? But either of them is adding something to hot path.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org