You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/05/03 01:03:18 UTC

[I] Heartbeat._check_heartbeat reports server down when only the socket close fails [texera]

Yicong-Huang opened a new issue, #4726:
URL: https://github.com/apache/texera/issues/4726

   ### What happened?
   
   `amber/src/main/python/core/runnables/heartbeat.py::Heartbeat._check_heartbeat` wraps both `socket.create_connection` and the subsequent `temp_socket.close()` in a single try/except. If `close()` raises after a successful connect, the method logs `"Server is down"` and returns `False` even though the JVM is reachable. The double-check in `run()` masks transient cases but the misclassification can still trigger unwarranted shutdowns.
   
   ### How to reproduce?
   
   ```python
   from unittest.mock import patch, MagicMock
   from threading import Event
   from core.runnables.heartbeat import Heartbeat
   
   hb = Heartbeat("localhost", 12345, 0.1, Event())
   sock = MagicMock()
   sock.close.side_effect = OSError("close failed")
   with patch("core.runnables.heartbeat.socket.create_connection", return_value=sock):
       print(hb._check_heartbeat())  # False, log says "Server is down"
   ```
   
   ### Version
   
   1.1.0-incubating (Pre-release/Master)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Heartbeat._check_heartbeat reports server down when only the socket close fails [texera]

Posted by "nathant27 (via GitHub)" <gi...@apache.org>.
nathant27 commented on issue #4726:
URL: https://github.com/apache/texera/issues/4726#issuecomment-4438631099

   /request-review @Yicong-Huang 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Heartbeat._check_heartbeat reports server down when only the socket close fails [texera]

Posted by "nathant27 (via GitHub)" <gi...@apache.org>.
nathant27 commented on issue #4726:
URL: https://github.com/apache/texera/issues/4726#issuecomment-4424148023

   /take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Heartbeat._check_heartbeat reports server down when only the socket close fails [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed issue #4726: Heartbeat._check_heartbeat reports server down when only the socket close fails
URL: https://github.com/apache/texera/issues/4726


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Heartbeat._check_heartbeat reports server down when only the socket close fails [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #4726:
URL: https://github.com/apache/texera/issues/4726#issuecomment-4438725709

   > /request-review @Yicong-Huang 
   
   This only works on PRs. 
   
   I am a bit busy on other tasks. Happy to revisit this later closely.
   
   
   For now I think the second one is an easy way to retry. Sometimes heartbeat can fail due to network instability, or other issues. Retry right away usually can reduce false alarms. So I suggest we keep it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Heartbeat._check_heartbeat reports server down when only the socket close fails [texera]

Posted by "nathant27 (via GitHub)" <gi...@apache.org>.
nathant27 commented on issue #4726:
URL: https://github.com/apache/texera/issues/4726#issuecomment-4426251167

   Hello. I plan on fixing this bug by replacing the old try except block with two separate try except blocks, and returning True as long as the connection successfully establishes. This would change the behavior a bit
   ```# in _check_heartbeat(self) new proposed version
   try:
               temp_socket = socket.create_connection(
                   (self._parsed_server_host, self._parsed_server_port), timeout=1
               )
           except Exception as e:
               logger.warning(f"Server is down with exception: {e}")
               return False
   
           try:
               temp_socket.close()
           except Exception as e:
               logger.warning(f"Failed to close socket: {e}")
           
           return True
   ```
   In my proposal, I changed the behavior of the function so that on successful connection it returns True(even if close raises exception), and returns False if socket.create_connection(...) raises. If close raises, we still return True but log something different in the logger.
   
   I changed it to this behavior because as stated 
   "If close() raises after a successful connect, the method logs "Server is down" and returns False even though the JVM is reachable. The double-check in run() masks transient cases but the misclassification can still trigger unwarranted shutdowns.", 
   which makes it seem like we want to avoid unwarranted shutdowns when .close() raises an exception, and the easiest way I thought of would be changing that behavior so that .close() raised exceptions still return True(assuming connection was established), to keep the run loop going.
   
   Was wondering if this was an appropriate way to handle the bug, or if we still want to keep the current behavior where an exception on close still returns false, which is the current behavior in the pinned test in amber/src/test/python/core/runnables/test_heartbeat.py, line 82
   test_returns_false_when_socket_close_raises


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Heartbeat._check_heartbeat reports server down when only the socket close fails [texera]

Posted by "nathant27 (via GitHub)" <gi...@apache.org>.
nathant27 commented on issue #4726:
URL: https://github.com/apache/texera/issues/4726#issuecomment-4438791983

   Sounds good! I'll work on this full implementation in my PR! I'll keep the second retry and keep current implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Heartbeat._check_heartbeat reports server down when only the socket close fails [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #4726:
URL: https://github.com/apache/texera/issues/4726#issuecomment-4438741509

   > Hello. I plan on fixing this bug by replacing the old try except block with two separate try except blocks, and returning True as long as the connection successfully establishes. This would change the behavior a bit.
   > proposed change
   > ```
   > try:
   >             temp_socket = socket.create_connection(
   >                 (self._parsed_server_host, self._parsed_server_port), timeout=1
   >             )
   >         except Exception as e:
   >             logger.warning(f"Server is down with exception: {e}")
   >             return False
   > 
   >         try:
   >             temp_socket.close()
   >         except Exception as e:
   >             logger.warning(f"Failed to close socket: {e}")
   >         
   >         return True
   > ```
   > In my proposal, I changed the behavior of the function so that on successful connection it returns True(even if close raises exception), and returns False if socket.create_connection(...) raises. If close raises, we still return True but log something different in the logger.
   > 
   > I changed it to this behavior because as stated 
   > "If close() raises after a successful connect, the method logs "Server is down" and returns False even though the JVM is reachable. The double-check in run() masks transient cases but the misclassification can still trigger unwarranted shutdowns.", 
   > which makes it seem like we want to avoid unwarranted shutdowns when .close() raises an exception, and the easiest way I thought of would be changing that behavior so that .close() raised exceptions still return True(assuming connection was established), to keep the run loop going.
   > 
   > This gets rid of the false negative case in the pinned test in amber/src/test/python/core/runnables/test_heartbeat.py, line 82
   > test_returns_false_when_socket_close_raises
   
   
   For this kind of code details, feel free to open PRs directly and we can use PRs to review. Use issues to discuss high level ideas, possibly before coding. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Heartbeat._check_heartbeat reports server down when only the socket close fails [texera]

Posted by "nathant27 (via GitHub)" <gi...@apache.org>.
nathant27 commented on issue #4726:
URL: https://github.com/apache/texera/issues/4726#issuecomment-4427305816

   Also, another question I have is on the behavior for Heartbeat.run
   
   ```
       @overrides
       def run(self) -> None:
           while not self._stop_event.wait(timeout=self._interval):
               alive = self._check_heartbeat()
               if not alive:
                   # double check
                   still_alive = self._check_heartbeat()
   
                   if not still_alive:
                       parent_pid = os.getppid()
   ```
   
   Is there a reason why we double check if not alive twice besides double checking for a close raising an exception and returning false in _check_heartbeat? If it's only for that purpose, can I remove the redundant if check?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org