Change in asterixdb[master]: [ASTERIXDB-3250][STO] Ensure writing indexes' empty files to S3

[AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>]

From Wail Alkowaileet <wa...@gmail.com>:

Wail Alkowaileet has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17737 )


Change subject: [ASTERIXDB-3250][STO] Ensure writing indexes' empty files to S3
......................................................................

[ASTERIXDB-3250][STO] Ensure writing indexes' empty files to S3

- user model changes: no
- storage format changes: no
- interface changes: yes

Details:
Empty indexes files are not written to S3.
This patch fixes this issue.

Interface changes:
- Add isEmpty() in ICloudBufferedWriter

Change-Id: I740c0d9e82bd0f933a37c4f68c3d95dc715cb46b
---
M asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/CloudResettableInputStream.java
M asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3BufferedWriter.java
M asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/ICloudBufferedWriter.java
3 files changed, 42 insertions(+), 2 deletions(-)



  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/37/17737/1

diff --git a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/CloudResettableInputStream.java b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/CloudResettableInputStream.java
index 3a7504a..7ba95c5 100644
--- a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/CloudResettableInputStream.java
+++ b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/CloudResettableInputStream.java
@@ -100,7 +100,13 @@
     public void finish() throws HyracksDataException {
         open();
         try {
-            if (writeBuffer.position() > 0) {
+            if (writeBuffer.position() > 0 || bufferedWriter.isEmpty()) {
+                /*
+                 * upload if:
+                 * (1) the writeBuffer is not empty
+                 * OR
+                 * (2) nothing was written to the file at all to ensure writing empty file
+                 */
                 uploadAndWait();
             }
             bufferedWriter.finish();
diff --git a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/ICloudBufferedWriter.java b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/ICloudBufferedWriter.java
index 4f08111..16e79f9 100644
--- a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/ICloudBufferedWriter.java
+++ b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/ICloudBufferedWriter.java
@@ -34,6 +34,13 @@
     int upload(InputStream stream, int length);
 
     /**
+     * Checks whether the writer has not written anything
+     *
+     * @return true if nothing was written, false otherwise
+     */
+    boolean isEmpty();
+
+    /**
      * Finishes the upload
      *
      * @throws HyracksDataException HyracksDataException
diff --git a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3BufferedWriter.java b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3BufferedWriter.java
index 71246f1..2efd214 100644
--- a/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3BufferedWriter.java
+++ b/asterixdb/asterix-cloud/src/main/java/org/apache/asterix/cloud/clients/aws/s3/S3BufferedWriter.java
@@ -73,10 +73,17 @@
     }
 
     @Override
+    public boolean isEmpty() {
+        return uploadId == null;
+    }
+
+    @Override
     public void finish() throws HyracksDataException {
         if (uploadId == null) {
-            return;
+            throw new IllegalStateException("Cannot finish without writing any bytes");
         }
+
+        // A non-empty files, proceed with completing the multipart upload
         CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder().parts(partQueue).build();
         CompleteMultipartUploadRequest completeMultipartUploadRequest = CompleteMultipartUploadRequest.builder()
                 .bucket(bucket).key(path).uploadId(uploadId).multipartUpload(completedMultipartUpload).build();

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17737
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I740c0d9e82bd0f933a37c4f68c3d95dc715cb46b
Gerrit-Change-Number: 17737
Gerrit-PatchSet: 1
Gerrit-Owner: Wail Alkowaileet <wa...@gmail.com>
Gerrit-MessageType: newchange