[GitHub] [incubator-druid] crodier commented on a change in pull request #5344: Fix races in LookupSnapshotTaker, CoordinatorPollingBasicAuthenticatorCacheManager

[GitBox <gi...@apache.org>]

crodier commented on a change in pull request #5344: Fix races in LookupSnapshotTaker, CoordinatorPollingBasicAuthenticatorCacheManager
URL: https://github.com/apache/incubator-druid/pull/5344#discussion_r299955603
 
 

 ##########
 File path: java-util/src/main/java/io/druid/java/util/common/FileUtils.java
 ##########
 @@ -150,15 +156,76 @@ public void addFile(File file)
    * }}</pre>
    *
    * @param file the file to map
+   *
    * @return a {@link MappedByteBufferHandler}, wrapping a read-only buffer reflecting {@code file}
-   * @throws FileNotFoundException if the {@code file} does not exist
-   * @throws IOException if an I/O error occurs
    *
+   * @throws FileNotFoundException if the {@code file} does not exist
+   * @throws IOException           if an I/O error occurs
    * @see FileChannel#map(FileChannel.MapMode, long, long)
    */
   public static MappedByteBufferHandler map(File file) throws IOException
   {
     MappedByteBuffer mappedByteBuffer = Files.map(file);
     return new MappedByteBufferHandler(mappedByteBuffer);
   }
+
+  /**
+   * Write to a file atomically, by first writing to a temporary file in the same directory and then moving it to
+   * the target location. This function attempts to clean up its temporary files when possible, but they may stick
+   * around (for example, if the JVM crashes partway through executing the function). In any case, the target file
+   * should be unharmed.
+   *
+   * The OutputStream passed to the consumer is uncloseable; calling close on it will do nothing. This is to ensure
+   * that the stream stays open so we can fsync it here before closing. Hopefully, this doesn't cause any problems
+   * for callers.
+   *
+   * This method is not just thread-safe, but is also safe to use from multiple processes on the same machine.
+   */
+  public static void writeAtomically(final File file, OutputStreamConsumer f) throws IOException
+  {
+    writeAtomically(file, file.getParentFile(), f);
+  }
+
+  private static void writeAtomically(final File file, final File tmpDir, OutputStreamConsumer f) throws IOException
+  {
+    final File tmpFile = new File(tmpDir, String.format(".%s.%s", file.getName(), UUID.randomUUID()));
+
+    try {
+      try (final FileOutputStream out = new FileOutputStream(tmpFile)) {
+        // Pass f an uncloseable stream so we can fsync before closing.
+        f.accept(uncloseable(out));
+
+        // fsync to avoid write-then-rename-then-crash causing empty files on some filesystems.
+        out.getChannel().force(true);
+      }
+
+      // No exception thrown; do the move.
+      java.nio.file.Files.move(
+          tmpFile.toPath(),
+          file.toPath(),
+          StandardCopyOption.ATOMIC_MOVE,
+          StandardCopyOption.REPLACE_EXISTING
+      );
+    }
+    finally {
+      tmpFile.delete();
+    }
+  }
+
+  private static OutputStream uncloseable(final OutputStream out) throws IOException
+  {
+    return new FilterOutputStream(out)
 
 Review comment:
   In unit tests locally with 15mb files, using a BufferedOutputStream here, takes 1 second vs. ~16 seconds.  FilterOutputStream isn't buffered, is too slow for practical use?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org