You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/03/04 19:14:50 UTC
[GitHub] [arrow] lidavidm opened a new pull request #9633: ARROW-11560: [C++][FlightRPC] fix mutex error on SIGINT
lidavidm opened a new pull request #9633:
URL: https://github.com/apache/arrow/pull/9633
Recently, interrupting a Python Flight server started aborting the process instead, because gRPC on Linux started using some code which is not signal-safe. This PR fixes that by spawning a separate thread that waits on a semaphore, then shuts down the server; the semaphore is notified by the signal handler (which is a signal-safe operation).
This is a little janky, but fixes the issue. This still builds on Windows in my tests, but I couldn't actually import the resulting PyArrow to test it. However, PyArrow 3.0 via Conda still works on Windows. I'm not able to test on MacOS myself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9633: ARROW-11560: [C++][FlightRPC] fix mutex error on SIGINT
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9633:
URL: https://github.com/apache/arrow/pull/9633#issuecomment-790919414
https://issues.apache.org/jira/browse/ARROW-11560
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #9633: ARROW-11560: [C++][FlightRPC] fix mutex error on SIGINT
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #9633:
URL: https://github.com/apache/arrow/pull/9633#issuecomment-792808918
Thanks for the comments. I've fixed up things here to use a self-pipe on all platforms.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on pull request #9633: ARROW-11560: [C++][FlightRPC] fix mutex error on SIGINT
Posted by GitBox <gi...@apache.org>.
cyb70289 commented on pull request #9633:
URL: https://github.com/apache/arrow/pull/9633#issuecomment-791150699
Also update PR comment to match latest code? `semaphore` -> `pipe`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed pull request #9633: ARROW-11560: [C++][FlightRPC] fix mutex error on SIGINT
Posted by GitBox <gi...@apache.org>.
pitrou closed pull request #9633:
URL: https://github.com/apache/arrow/pull/9633
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #9633: ARROW-11560: [C++][FlightRPC] fix mutex error on SIGINT
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #9633:
URL: https://github.com/apache/arrow/pull/9633#discussion_r588145650
##########
File path: cpp/src/arrow/flight/server.cc
##########
@@ -791,16 +799,30 @@ struct FlightServerBase::Impl {
void DoHandleSignal(int signum) {
got_signal_ = signum;
+#ifdef _WIN32
server_->Shutdown();
+#else
+ int saved_errno = errno;
+ // Ignore errors - pipe is nonblocking
+ write(self_pipe_[1], "0", 1);
+ errno = saved_errno;
+#endif
}
+
+#ifndef _WIN32
+ static void WaitForSignals(int fd) {
+ // Wait for a signal handler to write to the pipe
+ int8_t buf[1];
+ read(fd, /*buf=*/buf, /*count=*/1);
Review comment:
This could fail with `EINTR`. Of course, presumably that's not really an error here :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #9633: ARROW-11560: [C++][FlightRPC] fix mutex error on SIGINT
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #9633:
URL: https://github.com/apache/arrow/pull/9633#discussion_r588143037
##########
File path: cpp/src/arrow/flight/server.cc
##########
@@ -892,6 +914,25 @@ Status FlightServerBase::Serve() {
impl_->old_signal_handlers_.clear();
impl_->running_instance_ = impl_.get();
+#ifndef _WIN32
+ // Create the self-pipe for signal handlers to use
+ if (pipe(impl_->self_pipe_.data()) != 0) {
Review comment:
We have `CreatePipe` in `arrow/util/io_util.h`, and it does work on Windows.
##########
File path: cpp/src/arrow/flight/server.cc
##########
@@ -791,16 +799,30 @@ struct FlightServerBase::Impl {
void DoHandleSignal(int signum) {
got_signal_ = signum;
+#ifdef _WIN32
server_->Shutdown();
+#else
+ int saved_errno = errno;
+ // Ignore errors - pipe is nonblocking
+ write(self_pipe_[1], "0", 1);
+ errno = saved_errno;
+#endif
}
+
+#ifndef _WIN32
+ static void WaitForSignals(int fd) {
+ // Wait for a signal handler to write to the pipe
+ int8_t buf[1];
+ read(fd, /*buf=*/buf, /*count=*/1);
Review comment:
This could be fail with `EINTR`. Of course, presumably that's not really an error here :-)
##########
File path: cpp/src/arrow/flight/server.cc
##########
@@ -892,6 +914,25 @@ Status FlightServerBase::Serve() {
impl_->old_signal_handlers_.clear();
impl_->running_instance_ = impl_.get();
+#ifndef _WIN32
+ // Create the self-pipe for signal handlers to use
+ if (pipe(impl_->self_pipe_.data()) != 0) {
+ return arrow::internal::IOErrorFromErrno(
+ errno, "Could not initialize self-pipe to wait for signals");
+ }
+ // Make write end nonblocking
+ int flags = fcntl(impl_->self_pipe_[1], F_GETFL);
+ if (flags == -1) {
+ return arrow::internal::IOErrorFromErrno(
+ errno, "Could not initialize self-pipe to wait for signals");
+ }
+ flags |= O_NONBLOCK;
+ if (fcntl(impl_->self_pipe_[1], F_SETFL, flags) == -1) {
+ return arrow::internal::IOErrorFromErrno(
+ errno, "Could not initialize self-pipe to wait for signals");
+ }
+ std::thread handle_signals(&Impl::WaitForSignals, impl_->self_pipe_[0]);
Review comment:
Ideally this all would be factored in a RAII object (what happens if there is an error below?).
##########
File path: cpp/src/arrow/flight/server.cc
##########
@@ -908,7 +949,20 @@ Status FlightServerBase::Serve() {
RETURN_NOT_OK(
SetSignalHandler(impl_->signals_[i], impl_->old_signal_handlers_[i]).status());
}
-
+#ifndef _WIN32
+ // We don't care if writing is blocked - that means a signal was already raised
+ if (write(impl_->self_pipe_[1], "0", 1) < 0 && errno != EAGAIN &&
+ errno != EWOULDBLOCK && errno != EINTR) {
+ return arrow::internal::IOErrorFromErrno(errno, "Could not unblock signal thread");
+ }
+ if (close(impl_->self_pipe_[0]) < 0) {
Review comment:
`FileClose`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #9633: ARROW-11560: [C++][FlightRPC] fix mutex error on SIGINT
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #9633:
URL: https://github.com/apache/arrow/pull/9633#discussion_r587752941
##########
File path: cpp/src/arrow/flight/server.cc
##########
@@ -899,6 +921,13 @@ Status FlightServerBase::Serve() {
ARROW_ASSIGN_OR_RAISE(old_handler, SetSignalHandler(signum, new_handler));
impl_->old_signal_handlers_.push_back(std::move(old_handler));
}
+#ifndef _WIN32
+ if (sem_init(&impl_->waiting_sem_, /*pshared=*/0, /*value=*/0) != 0) {
Review comment:
Unfortunately I don't think macOS supports anonymous semaphores.
An alternative would be to use a non-blocking anonymous pipe. `write` is async-signal-safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org