You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/03/29 12:52:51 UTC

[GitHub] [hadoop] steveloughran commented on a change in pull request #4115: HADOOP-18180. Replace use of twitter util-core with java futures

steveloughran commented on a change in pull request #4115:
URL: https://github.com/apache/hadoop/pull/4115#discussion_r837419830



##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingBlockManager.java
##########
@@ -22,7 +22,7 @@
 import java.io.IOException;
 import java.nio.ByteBuffer;
 
-import com.twitter.util.FuturePool;
+import org.apache.hadoop.fs.common.ExecutorServiceFuturePool;

Review comment:
       nit, can you move to the apache import block

##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/CachingBlockManager.java
##########
@@ -354,7 +355,7 @@ private void readBlock(BufferData data, boolean isPrefetch, BufferData.State...
     }
 
     @Override
-    public Void applyE() {
+    public Void get() {

Review comment:
       we need to think what we want to do here.
   
   1. we would want errors to be counted and included in stream stats
   1. but we've seen abfs apps flooded with stack traces when a transient network error breaks every prefetch
   
   * its good to be aware of transient errors, but not worry too much until the final reaqd.
   
   created https://issues.apache.org/jira/browse/HADOOP-18185
   
   

##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/TestExecutorServiceFuturePool.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.common;
+
+import org.junit.Test;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+public class TestExecutorServiceFuturePool {
+  @Test
+  public void testRunnableSucceeds() throws Exception {
+    ExecutorService executorService = Executors.newFixedThreadPool(3);
+    try {
+      ExecutorServiceFuturePool futurePool = new ExecutorServiceFuturePool(executorService);
+      final AtomicBoolean atomicBoolean = new AtomicBoolean(false);
+      Future<Void> future = futurePool.executeRunnable(() -> atomicBoolean.set(true));
+      future.get(30, TimeUnit.SECONDS);
+      assertTrue(atomicBoolean.get());

Review comment:
       yeah, all assert trues need an error text to read when yetus fails. feel free to use AssertJ here if that helps...we are adopting it now

##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/read/TestS3InputStream.java
##########
@@ -24,8 +24,7 @@
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 
-import com.twitter.util.ExecutorServiceFuturePool;
-import com.twitter.util.FuturePool;
+import org.apache.hadoop.fs.common.ExecutorServiceFuturePool;

Review comment:
       nit; move into apache block

##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/TestExecutorServiceFuturePool.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.common;
+
+import org.junit.Test;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+public class TestExecutorServiceFuturePool {
+  @Test
+  public void testRunnableSucceeds() throws Exception {
+    ExecutorService executorService = Executors.newFixedThreadPool(3);

Review comment:
       as this is used everywhere, better to make a field, init in setup and in teardown, if non null, shut it down

##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
##########
@@ -76,8 +76,7 @@
 import com.amazonaws.services.s3.transfer.model.UploadResult;
 import com.amazonaws.event.ProgressListener;
 
-import com.twitter.util.ExecutorServiceFuturePool;
-import com.twitter.util.FuturePool;
+import org.apache.hadoop.fs.common.ExecutorServiceFuturePool;

Review comment:
       can you move to the apache import block (we try to keep imports under control...)

##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/TestExecutorServiceFuturePool.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.common;
+
+import org.junit.Test;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+public class TestExecutorServiceFuturePool {

Review comment:
       can you extend `AbstractHadoopTestBase` for the thread names, timeout etc.

##########
File path: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/TestExecutorServiceFuturePool.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.common;
+
+import org.junit.Test;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+public class TestExecutorServiceFuturePool {
+  @Test
+  public void testRunnableSucceeds() throws Exception {
+    ExecutorService executorService = Executors.newFixedThreadPool(3);
+    try {
+      ExecutorServiceFuturePool futurePool = new ExecutorServiceFuturePool(executorService);
+      final AtomicBoolean atomicBoolean = new AtomicBoolean(false);
+      Future<Void> future = futurePool.executeRunnable(() -> atomicBoolean.set(true));
+      future.get(30, TimeUnit.SECONDS);
+      assertTrue(atomicBoolean.get());
+    } finally {
+      executorService.shutdownNow();
+    }
+  }
+
+  @Test
+  public void testSupplierSucceeds() throws Exception {
+    ExecutorService executorService = Executors.newFixedThreadPool(3);
+    try {
+      ExecutorServiceFuturePool futurePool = new ExecutorServiceFuturePool(executorService);
+      final AtomicBoolean atomicBoolean = new AtomicBoolean(false);
+      Future<Void> future = futurePool.executeFunction(() -> {
+        atomicBoolean.set(true);
+        return null;
+      });
+      future.get(30, TimeUnit.SECONDS);
+      assertTrue(atomicBoolean.get());
+    } finally {
+      executorService.shutdownNow();
+    }
+  }
+
+  @Test
+  public void testRunnableFails() throws Exception {
+    ExecutorService executorService = Executors.newFixedThreadPool(3);
+    try {
+      ExecutorServiceFuturePool futurePool = new ExecutorServiceFuturePool(executorService);
+      Future<Void> future = futurePool.executeRunnable(() -> {
+        throw new IllegalStateException("deliberate");
+      });
+      assertThrows(ExecutionException.class, () -> future.get(30, TimeUnit.SECONDS));
+    } finally {
+      executorService.shutdownNow();
+    }
+  }
+
+  @Test
+  public void testSupplierFails() throws Exception {
+    ExecutorService executorService = Executors.newFixedThreadPool(3);
+    try {
+      ExecutorServiceFuturePool futurePool = new ExecutorServiceFuturePool(executorService);
+      Future<Void> future = futurePool.executeFunction(() -> {
+        throw new IllegalStateException("deliberate");
+      });
+      assertThrows(ExecutionException.class, () -> future.get(30, TimeUnit.SECONDS));

Review comment:
       prefer LambdaTestUtils intercept, which includes the toString value of the operation on a failure.

##########
File path: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/ExecutorServiceFuturePool.java
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.common;
+
+import java.util.Locale;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.function.Supplier;
+
+/**
+ * A FuturePool implementation backed by a java.util.concurrent.ExecutorService.
+ *
+ * If a piece of work has started, it cannot (currently) be cancelled.
+ *
+ * This class is a simplified version of <code>com.twitter:util-core_2.11</code> ExecutorServiceFuturePool
+ * designed to avoid depending on that Scala library. One problem with using a Scala library is that many
+ * downstream projects (eg Apache Spark) use Scala and they might want to use a different version of Scala
+ * from the version that Hadoop chooses to use.
+ *
+ */
+public class ExecutorServiceFuturePool {
+  private ExecutorService executor;
+
+  public ExecutorServiceFuturePool(ExecutorService executor) {

Review comment:
       interesting q here about what service to support/integrate with here, especially for faireness in processs with many readers.
   
   created https://issues.apache.org/jira/browse/HADOOP-18186 as a followup to this.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org