You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/04 02:44:15 UTC

[GitHub] [arrow] westonpace opened a new pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

westonpace opened a new pull request #11084:
URL: https://github.com/apache/arrow/pull/11084


   This PR adds two new utilities.
   
   The first is an asynchronous smart pointer which makes it easier to ensure that asynchronous tasks are finished before an object is destroyed (and generally makes it safe to capture `this`)
   
   The second is an asynchronous task group which collects futures and can help to ensure that all futures are completed.  It is similar to AllComplete except it doesn't require collecting all the futures at once.
   
   Combined, these two things can help give structured concurrency / nursery type control over asynchronous operations.  I have used these utilities in #10955 if you would like to see an example of them in action.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11084:
URL: https://github.com/apache/arrow/pull/11084#discussion_r704875386



##########
File path: cpp/src/arrow/util/async_util.h
##########
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/future.h"
+#include "arrow/util/mutex.h"
+
+namespace arrow {
+namespace util {
+
+/// Custom deleter for AsyncDestroyable objects
+template <typename T>
+struct DestroyingDeleter {
+  void operator()(T* p) { p->Destroy(); }
+};
+
+/// An object which should be asynchronously closed before it is destroyed
+///
+/// Classes can extend this to ensure that the close method is called and completed
+/// before the instance is deleted.  This provides smart_ptr / delete semantics for
+/// objects with an asynchronous destructor.
+///
+/// Classes which extend this must be constructed using MakeSharedAsync or MakeUniqueAsync
+class ARROW_EXPORT AsyncDestroyable {
+ public:
+  AsyncDestroyable();
+  virtual ~AsyncDestroyable();
+
+  /// A future which will complete when the AsyncDestroyable has finished and is ready
+  /// to be deleted.
+  ///
+  /// This can be used to ensure all work done by this object has been completed before
+  /// proceeding.
+  Future<> on_closed() { return on_closed_; }
+
+ protected:
+  /// Subclasses should override this and perform any cleanup.  Once the future returned
+  /// by this method finishes then this object is eligible for destruction and any
+  /// reference to `this` will be invalid
+  virtual Future<> DoDestroy() = 0;
+
+ private:
+  void Destroy();
+
+  Future<> on_closed_;
+#ifndef NDEBUG
+  bool constructed_correctly_ = false;
+#endif
+
+  template <typename T>
+  friend struct DestroyingDeleter;
+  template <typename T, typename... Args>
+  friend std::shared_ptr<T> MakeSharedAsync(Args&&... args);
+  template <typename T, typename... Args>
+  friend std::unique_ptr<T, DestroyingDeleter<T>> MakeUniqueAsync(Args&&... args);
+};
+
+template <typename T, typename... Args>
+std::shared_ptr<T> MakeSharedAsync(Args&&... args) {
+  static_assert(std::is_base_of<AsyncDestroyable, T>::value,
+                "Nursery::MakeSharedCloseable only works with AsyncDestroyable types");
+  std::shared_ptr<T> ptr(new T(std::forward<Args&&>(args)...), DestroyingDeleter<T>());
+#ifndef NDEBUG
+  ptr->constructed_correctly_ = true;
+#endif
+  return ptr;
+}
+
+template <typename T, typename... Args>
+std::unique_ptr<T, DestroyingDeleter<T>> MakeUniqueAsync(Args&&... args) {
+  static_assert(std::is_base_of<AsyncDestroyable, T>::value,
+                "Nursery::MakeUniqueCloseable only works with AsyncDestroyable types");
+  std::unique_ptr<T, DestroyingDeleter<T>> ptr(new T(std::forward<Args>(args)...),
+                                               DestroyingDeleter<T>());
+#ifndef NDEBUG
+  ptr->constructed_correctly_ = true;
+#endif
+  return ptr;
+}
+
+/// A utility which keeps track of a collection of asynchronous tasks
+///
+/// This can be used to provide structured concurrency for asynchronous development.
+/// A task group created at a high level can be distributed amongst low level components
+/// which register work to be completed.  The high level job can then wait for all work
+/// to be completed before cleaning up.
+class AsyncTaskGroup {

Review comment:
       Ah, thanks.  Added.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #11084:
URL: https://github.com/apache/arrow/pull/11084#issuecomment-915681730


   I've moved this back to ready for review.  AsyncTaskGroup is similar but not the same as the TaskGroups.  SerialTaskGroup and ThreadedTaskGroup take care of scheduling the task (submitting it to the executor) as well as tracking the task.  The AsyncTaskGroup is only responsible for tracking a set of tasks that were scheduled elsewhere.  ThreadedTaskGroup could maybe use AsyncTaskGroup through composition but that could be done elsewhere.
   
   The scheduling in the dataset writer case happens in the file queue which doesn't quite fit the SerialTaskGroup / ThreadedTaskGroup model because each task should run serially (like SerialTaskGroup) but asynchronously (like ThreadedTaskGroup).  I could potentially extract the logic in DatasetWriterFileQueue into a AsyncSerialTaskGroup but for the moment I think what this PR and the dataset writer PR proposes is fine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on a change in pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11084:
URL: https://github.com/apache/arrow/pull/11084#discussion_r702892278



##########
File path: cpp/src/arrow/util/async_util.h
##########
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/future.h"
+#include "arrow/util/mutex.h"
+
+namespace arrow {
+namespace util {
+
+/// Custom deleter for AsyncDestroyable objects
+template <typename T>
+struct DestroyingDeleter {
+  void operator()(T* p) { p->Destroy(); }
+};
+
+/// An object which should be asynchronously closed before it is destroyed
+///
+/// Classes can extend this to ensure that the close method is called and completed
+/// before the instance is deleted.  This provides smart_ptr / delete semantics for
+/// objects with an asynchronous destructor.
+///
+/// Classes which extend this must be constructed using MakeSharedAsync or MakeUniqueAsync
+class ARROW_EXPORT AsyncDestroyable {
+ public:
+  AsyncDestroyable();
+  virtual ~AsyncDestroyable();
+
+  /// A future which will complete when the AsyncDestroyable has finished and is ready
+  /// to be deleted.
+  ///
+  /// This can be used to ensure all work done by this object has been completed before
+  /// proceeding.
+  Future<> on_closed() { return on_closed_; }
+
+ protected:
+  /// Subclasses should override this and perform any cleanup.  Once the future returned
+  /// by this method finishes then this object is eligible for destruction and any
+  /// reference to `this` will be invalid
+  virtual Future<> DoDestroy() = 0;
+
+ private:
+  void Destroy();
+
+  Future<> on_closed_;
+#ifndef NDEBUG
+  bool constructed_correctly_ = false;
+#endif
+
+  template <typename T>
+  friend struct DestroyingDeleter;
+  template <typename T, typename... Args>
+  friend std::shared_ptr<T> MakeSharedAsync(Args&&... args);
+  template <typename T, typename... Args>
+  friend std::unique_ptr<T, DestroyingDeleter<T>> MakeUniqueAsync(Args&&... args);
+};
+
+template <typename T, typename... Args>
+std::shared_ptr<T> MakeSharedAsync(Args&&... args) {
+  static_assert(std::is_base_of<AsyncDestroyable, T>::value,
+                "Nursery::MakeSharedCloseable only works with AsyncDestroyable types");
+  std::shared_ptr<T> ptr(new T(std::forward<Args&&>(args)...), DestroyingDeleter<T>());
+#ifndef NDEBUG
+  ptr->constructed_correctly_ = true;
+#endif
+  return ptr;
+}
+
+template <typename T, typename... Args>
+std::unique_ptr<T, DestroyingDeleter<T>> MakeUniqueAsync(Args&&... args) {
+  static_assert(std::is_base_of<AsyncDestroyable, T>::value,
+                "Nursery::MakeUniqueCloseable only works with AsyncDestroyable types");
+  std::unique_ptr<T, DestroyingDeleter<T>> ptr(new T(std::forward<Args>(args)...),
+                                               DestroyingDeleter<T>());
+#ifndef NDEBUG
+  ptr->constructed_correctly_ = true;
+#endif
+  return ptr;
+}
+
+/// A utility which keeps track of a collection of asynchronous tasks
+///
+/// This can be used to provide structured concurrency for asynchronous development.
+/// A task group created at a high level can be distributed amongst low level components
+/// which register work to be completed.  The high level job can then wait for all work
+/// to be completed before cleaning up.
+class AsyncTaskGroup {

Review comment:
       Ah, MSVC isn't happy since this needs ARROW_EXPORT too.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

Posted by GitBox <gi...@apache.org>.
westonpace commented on pull request #11084:
URL: https://github.com/apache/arrow/pull/11084#issuecomment-912906107


   Drafting this because I want to take a quick look to see if I can reuse the existing task group or if the semantics aren't quite right.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11084:
URL: https://github.com/apache/arrow/pull/11084#discussion_r704875302



##########
File path: cpp/src/arrow/util/async_util_test.cc
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/async_util.h"
+
+#include <gtest/gtest.h>
+
+#include "arrow/result.h"
+#include "arrow/testing/future_util.h"
+#include "arrow/testing/gtest_util.h"
+
+namespace arrow {
+namespace util {
+
+class GatingDestroyable : public AsyncDestroyable {
+ public:
+  GatingDestroyable(Future<> close_future, bool* destroyed)
+      : close_future_(std::move(close_future)), destroyed_(destroyed) {}
+  ~GatingDestroyable() override { *destroyed_ = true; }
+
+ protected:
+  Future<> DoDestroy() override { return close_future_; }
+
+ private:
+  Future<> close_future_;
+  bool* destroyed_;
+};
+
+template <typename Factory>
+void TestAsyncDestroyable(Factory factory) {
+  Future<> gate = Future<>::Make();
+  bool destroyed = false;
+  bool on_closed = false;
+  {
+    auto obj = factory(gate, &destroyed);
+    obj->on_closed().AddCallback([&](const Status& st) { on_closed = true; });
+    ASSERT_FALSE(destroyed);
+  }
+  ASSERT_FALSE(destroyed);
+  ASSERT_FALSE(on_closed);
+  gate.MarkFinished();
+  ASSERT_TRUE(destroyed);
+  ASSERT_TRUE(on_closed);
+}
+
+TEST(AsyncDestroyable, MakeShared) {
+  TestAsyncDestroyable([](Future<> gate, bool* destroyed) {
+    return MakeSharedAsync<GatingDestroyable>(gate, destroyed);
+  });
+}
+
+TEST(AsyncDestroyable, MakeUnique) {
+  Future<> gate = Future<>::Make();
+  bool destroyed = false;
+  bool on_closed = false;
+  {
+    std::unique_ptr<GatingDestroyable, DestroyingDeleter<GatingDestroyable>> obj =
+        MakeUniqueAsync<GatingDestroyable>(gate, &destroyed);
+    obj->on_closed().AddCallback([&](const Status& st) { on_closed = true; });
+    ASSERT_FALSE(destroyed);
+  }
+  ASSERT_FALSE(destroyed);
+  ASSERT_FALSE(on_closed);
+  gate.MarkFinished();
+  ASSERT_TRUE(destroyed);
+  ASSERT_TRUE(on_closed);
+}
+
+TEST(AsyncTaskGroup, Basic) {
+  AsyncTaskGroup task_group;
+  Future<> fut1 = Future<>::Make();
+  Future<> fut2 = Future<>::Make();
+  ASSERT_OK(task_group.AddTask(fut1));
+  ASSERT_OK(task_group.AddTask(fut2));
+  Future<> all_done = task_group.WaitForTasksToFinish();
+  AssertNotFinished(all_done);
+  fut1.MarkFinished();
+  AssertNotFinished(all_done);
+  fut2.MarkFinished();
+  ASSERT_FINISHES_OK(all_done);
+}
+
+TEST(AsyncTaskGroup, NoTasks) {
+  AsyncTaskGroup task_group;
+  ASSERT_FINISHES_OK(task_group.WaitForTasksToFinish());
+}
+
+TEST(AsyncTaskGroup, AddAfterDone) {
+  AsyncTaskGroup task_group;
+  ASSERT_FINISHES_OK(task_group.WaitForTasksToFinish());
+  ASSERT_RAISES(Invalid, task_group.AddTask(Future<>::Make()));
+}
+
+TEST(AsyncTaskGroup, AddAfterWaitButBeforeFinish) {
+  AsyncTaskGroup task_group;
+  Future<> task_one = Future<>::Make();
+  ASSERT_OK(task_group.AddTask(task_one));
+  Future<> finish_fut = task_group.WaitForTasksToFinish();
+  AssertNotFinished(finish_fut);
+  Future<> task_two = Future<>::Make();
+  ASSERT_OK(task_group.AddTask(task_two));
+  AssertNotFinished(finish_fut);
+  task_one.MarkFinished();
+  AssertNotFinished(finish_fut);
+  task_two.MarkFinished();
+  AssertFinished(finish_fut);
+  ASSERT_FINISHES_OK(finish_fut);
+}
+
+TEST(AsyncTaskGroup, Error) {
+  AsyncTaskGroup task_group;
+  Future<> failed_task = Future<>::MakeFinished(Status::Invalid("XYZ"));
+  ASSERT_OK(task_group.AddTask(failed_task));
+  ASSERT_FINISHES_AND_RAISES(Invalid, task_group.WaitForTasksToFinish());
+}
+
+TEST(AsyncTaskGroup, TaskFinishesAfterError) {
+  AsyncTaskGroup task_group;
+  Future<> fut1 = Future<>::Make();
+  ASSERT_OK(task_group.AddTask(fut1));
+  ASSERT_OK(task_group.AddTask(Future<>::MakeFinished(Status::Invalid("XYZ"))));
+  Future<> finished_fut = task_group.WaitForTasksToFinish();
+  AssertNotFinished(finished_fut);
+  fut1.MarkFinished();
+  ASSERT_FINISHES_AND_RAISES(Invalid, finished_fut);
+}
+
+TEST(AsyncTaskGroup, AddAfterFailed) {
+  AsyncTaskGroup task_group;
+  ASSERT_OK(task_group.AddTask(Future<>::MakeFinished(Status::Invalid("XYZ"))));
+  ASSERT_RAISES(Invalid, task_group.AddTask(Future<>::Make()));
+  ASSERT_FINISHES_AND_RAISES(Invalid, task_group.WaitForTasksToFinish());
+}
+

Review comment:
       I added a FailsAfterAdd test.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm closed pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #11084:
URL: https://github.com/apache/arrow/pull/11084


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace edited a comment on pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

Posted by GitBox <gi...@apache.org>.
westonpace edited a comment on pull request #11084:
URL: https://github.com/apache/arrow/pull/11084#issuecomment-915681730


   I've moved this back to ready for review.  AsyncTaskGroup is similar but not the same as the TaskGroups.  SerialTaskGroup and ThreadedTaskGroup take care of scheduling the task (submitting it to the executor) as well as tracking the task.  The AsyncTaskGroup is only responsible for tracking a set of tasks that were scheduled elsewhere.  ThreadedTaskGroup could maybe use AsyncTaskGroup through composition but that could be done in a future PR if needed.
   
   The scheduling in the dataset writer case happens in the file queue which doesn't quite fit the SerialTaskGroup / ThreadedTaskGroup model because each task should run serially (like SerialTaskGroup) but asynchronously (like ThreadedTaskGroup).  I could potentially extract the logic in DatasetWriterFileQueue into a AsyncSerialTaskGroup but for the moment I think what this PR and the dataset writer PR proposes is fine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] lidavidm commented on a change in pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

Posted by GitBox <gi...@apache.org>.
lidavidm commented on a change in pull request #11084:
URL: https://github.com/apache/arrow/pull/11084#discussion_r702888612



##########
File path: cpp/src/arrow/util/async_util_test.cc
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/async_util.h"
+
+#include <gtest/gtest.h>
+
+#include "arrow/result.h"
+#include "arrow/testing/future_util.h"
+#include "arrow/testing/gtest_util.h"
+
+namespace arrow {
+namespace util {
+
+class GatingDestroyable : public AsyncDestroyable {
+ public:
+  GatingDestroyable(Future<> close_future, bool* destroyed)
+      : close_future_(std::move(close_future)), destroyed_(destroyed) {}
+  ~GatingDestroyable() override { *destroyed_ = true; }
+
+ protected:
+  Future<> DoDestroy() override { return close_future_; }
+
+ private:
+  Future<> close_future_;
+  bool* destroyed_;
+};
+
+template <typename Factory>
+void TestAsyncDestroyable(Factory factory) {
+  Future<> gate = Future<>::Make();
+  bool destroyed = false;
+  bool on_closed = false;
+  {
+    auto obj = factory(gate, &destroyed);
+    obj->on_closed().AddCallback([&](const Status& st) { on_closed = true; });
+    ASSERT_FALSE(destroyed);
+  }
+  ASSERT_FALSE(destroyed);
+  ASSERT_FALSE(on_closed);
+  gate.MarkFinished();
+  ASSERT_TRUE(destroyed);
+  ASSERT_TRUE(on_closed);
+}
+
+TEST(AsyncDestroyable, MakeShared) {
+  TestAsyncDestroyable([](Future<> gate, bool* destroyed) {
+    return MakeSharedAsync<GatingDestroyable>(gate, destroyed);
+  });
+}
+
+TEST(AsyncDestroyable, MakeUnique) {

Review comment:
       Why doesn't this use TestAsyncDestroyable?

##########
File path: cpp/src/arrow/util/async_util_test.cc
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/async_util.h"
+
+#include <gtest/gtest.h>
+
+#include "arrow/result.h"
+#include "arrow/testing/future_util.h"
+#include "arrow/testing/gtest_util.h"
+
+namespace arrow {
+namespace util {
+
+class GatingDestroyable : public AsyncDestroyable {
+ public:
+  GatingDestroyable(Future<> close_future, bool* destroyed)
+      : close_future_(std::move(close_future)), destroyed_(destroyed) {}
+  ~GatingDestroyable() override { *destroyed_ = true; }
+
+ protected:
+  Future<> DoDestroy() override { return close_future_; }
+
+ private:
+  Future<> close_future_;
+  bool* destroyed_;
+};
+
+template <typename Factory>
+void TestAsyncDestroyable(Factory factory) {
+  Future<> gate = Future<>::Make();
+  bool destroyed = false;
+  bool on_closed = false;
+  {
+    auto obj = factory(gate, &destroyed);
+    obj->on_closed().AddCallback([&](const Status& st) { on_closed = true; });
+    ASSERT_FALSE(destroyed);
+  }
+  ASSERT_FALSE(destroyed);
+  ASSERT_FALSE(on_closed);
+  gate.MarkFinished();
+  ASSERT_TRUE(destroyed);
+  ASSERT_TRUE(on_closed);
+}
+
+TEST(AsyncDestroyable, MakeShared) {
+  TestAsyncDestroyable([](Future<> gate, bool* destroyed) {
+    return MakeSharedAsync<GatingDestroyable>(gate, destroyed);
+  });
+}
+
+TEST(AsyncDestroyable, MakeUnique) {
+  Future<> gate = Future<>::Make();
+  bool destroyed = false;
+  bool on_closed = false;
+  {
+    std::unique_ptr<GatingDestroyable, DestroyingDeleter<GatingDestroyable>> obj =
+        MakeUniqueAsync<GatingDestroyable>(gate, &destroyed);
+    obj->on_closed().AddCallback([&](const Status& st) { on_closed = true; });
+    ASSERT_FALSE(destroyed);
+  }
+  ASSERT_FALSE(destroyed);
+  ASSERT_FALSE(on_closed);
+  gate.MarkFinished();
+  ASSERT_TRUE(destroyed);
+  ASSERT_TRUE(on_closed);
+}
+
+TEST(AsyncTaskGroup, Basic) {
+  AsyncTaskGroup task_group;
+  Future<> fut1 = Future<>::Make();
+  Future<> fut2 = Future<>::Make();
+  ASSERT_OK(task_group.AddTask(fut1));
+  ASSERT_OK(task_group.AddTask(fut2));
+  Future<> all_done = task_group.WaitForTasksToFinish();
+  AssertNotFinished(all_done);
+  fut1.MarkFinished();
+  AssertNotFinished(all_done);
+  fut2.MarkFinished();
+  ASSERT_FINISHES_OK(all_done);
+}
+
+TEST(AsyncTaskGroup, NoTasks) {
+  AsyncTaskGroup task_group;
+  ASSERT_FINISHES_OK(task_group.WaitForTasksToFinish());
+}
+
+TEST(AsyncTaskGroup, AddAfterDone) {
+  AsyncTaskGroup task_group;
+  ASSERT_FINISHES_OK(task_group.WaitForTasksToFinish());
+  ASSERT_RAISES(Invalid, task_group.AddTask(Future<>::Make()));
+}
+
+TEST(AsyncTaskGroup, AddAfterWaitButBeforeFinish) {
+  AsyncTaskGroup task_group;
+  Future<> task_one = Future<>::Make();
+  ASSERT_OK(task_group.AddTask(task_one));
+  Future<> finish_fut = task_group.WaitForTasksToFinish();
+  AssertNotFinished(finish_fut);
+  Future<> task_two = Future<>::Make();
+  ASSERT_OK(task_group.AddTask(task_two));
+  AssertNotFinished(finish_fut);
+  task_one.MarkFinished();
+  AssertNotFinished(finish_fut);
+  task_two.MarkFinished();
+  AssertFinished(finish_fut);
+  ASSERT_FINISHES_OK(finish_fut);
+}
+
+TEST(AsyncTaskGroup, Error) {
+  AsyncTaskGroup task_group;
+  Future<> failed_task = Future<>::MakeFinished(Status::Invalid("XYZ"));
+  ASSERT_OK(task_group.AddTask(failed_task));
+  ASSERT_FINISHES_AND_RAISES(Invalid, task_group.WaitForTasksToFinish());
+}
+
+TEST(AsyncTaskGroup, TaskFinishesAfterError) {
+  AsyncTaskGroup task_group;
+  Future<> fut1 = Future<>::Make();
+  ASSERT_OK(task_group.AddTask(fut1));
+  ASSERT_OK(task_group.AddTask(Future<>::MakeFinished(Status::Invalid("XYZ"))));
+  Future<> finished_fut = task_group.WaitForTasksToFinish();
+  AssertNotFinished(finished_fut);
+  fut1.MarkFinished();
+  ASSERT_FINISHES_AND_RAISES(Invalid, finished_fut);
+}
+
+TEST(AsyncTaskGroup, AddAfterFailed) {
+  AsyncTaskGroup task_group;
+  ASSERT_OK(task_group.AddTask(Future<>::MakeFinished(Status::Invalid("XYZ"))));
+  ASSERT_RAISES(Invalid, task_group.AddTask(Future<>::Make()));
+  ASSERT_FINISHES_AND_RAISES(Invalid, task_group.WaitForTasksToFinish());
+}
+

Review comment:
       Since AddTask special-cases adding completed futures, maybe for completeness we should have a test of marking a future failed after it's been added to the group as well.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] westonpace commented on a change in pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11084:
URL: https://github.com/apache/arrow/pull/11084#discussion_r704875172



##########
File path: cpp/src/arrow/util/async_util_test.cc
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/util/async_util.h"
+
+#include <gtest/gtest.h>
+
+#include "arrow/result.h"
+#include "arrow/testing/future_util.h"
+#include "arrow/testing/gtest_util.h"
+
+namespace arrow {
+namespace util {
+
+class GatingDestroyable : public AsyncDestroyable {
+ public:
+  GatingDestroyable(Future<> close_future, bool* destroyed)
+      : close_future_(std::move(close_future)), destroyed_(destroyed) {}
+  ~GatingDestroyable() override { *destroyed_ = true; }
+
+ protected:
+  Future<> DoDestroy() override { return close_future_; }
+
+ private:
+  Future<> close_future_;
+  bool* destroyed_;
+};
+
+template <typename Factory>
+void TestAsyncDestroyable(Factory factory) {
+  Future<> gate = Future<>::Make();
+  bool destroyed = false;
+  bool on_closed = false;
+  {
+    auto obj = factory(gate, &destroyed);
+    obj->on_closed().AddCallback([&](const Status& st) { on_closed = true; });
+    ASSERT_FALSE(destroyed);
+  }
+  ASSERT_FALSE(destroyed);
+  ASSERT_FALSE(on_closed);
+  gate.MarkFinished();
+  ASSERT_TRUE(destroyed);
+  ASSERT_TRUE(on_closed);
+}
+
+TEST(AsyncDestroyable, MakeShared) {
+  TestAsyncDestroyable([](Future<> gate, bool* destroyed) {
+    return MakeSharedAsync<GatingDestroyable>(gate, destroyed);
+  });
+}
+
+TEST(AsyncDestroyable, MakeUnique) {

Review comment:
       Must've gotten lost in the switches.  That was originally the plan and I've switched back to it.  Thanks for catching this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] github-actions[bot] commented on pull request #11084: ARROW-13680: [C++] Create an asynchronous nursery to simplify capture logic

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11084:
URL: https://github.com/apache/arrow/pull/11084#issuecomment-912894715


   https://issues.apache.org/jira/browse/ARROW-13680


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org