You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/10/19 06:12:35 UTC

[GitHub] [iceberg] kbendick commented on a change in pull request #3288: Aliyun: OSS outputstream implementation

kbendick commented on a change in pull request #3288:
URL: https://github.com/apache/iceberg/pull/3288#discussion_r731519099



##########
File path: aliyun/src/test/java/org/apache/iceberg/aliyun/oss/TestOSSOutputStream.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.aliyun.oss;
+
+import com.aliyun.oss.OSS;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Random;
+import java.util.UUID;
+import org.apache.commons.io.IOUtils;
+import org.apache.iceberg.aliyun.AliyunProperties;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.mockito.AdditionalAnswers.delegatesTo;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.reset;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+public class TestOSSOutputStream extends AliyunOSSTestBase {
+  private static final Logger LOG = LoggerFactory.getLogger(TestOSSOutputStream.class);
+
+  private final OSS oss = oss().get();

Review comment:
       Would it make sense to possibly name the client in `AliyuanOSSTestBase` from `oss` to something like `ossClient`?
   
   Seeing `oss = oss().get()` is weird for me. Is this a common pattern with the `AliyunOSSTestUtility` library? I must say it looks quite handy to have available.

##########
File path: aliyun/src/test/java/org/apache/iceberg/aliyun/oss/AliyunOSSTestRule.java
##########
@@ -55,6 +55,18 @@ public void evaluate() throws Throwable {
     };
   }
 
+  /**
+   * Returns the common key prefix for those newly created objects in test cases. For example, we set the test bucket
+   * to be 'oss-testing-bucket' and the key prefix to be 'iceberg-objects/', then the produced objects in test cases
+   * will be:
+   * <pre>

Review comment:
       👍 

##########
File path: aliyun/src/test/java/org/apache/iceberg/aliyun/oss/TestOSSOutputStream.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.aliyun.oss;
+
+import com.aliyun.oss.OSS;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Random;
+import java.util.UUID;
+import org.apache.commons.io.IOUtils;
+import org.apache.iceberg.aliyun.AliyunProperties;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.mockito.AdditionalAnswers.delegatesTo;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.reset;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+public class TestOSSOutputStream extends AliyunOSSTestBase {
+  private static final Logger LOG = LoggerFactory.getLogger(TestOSSOutputStream.class);
+
+  private final OSS oss = oss().get();
+  private final OSS ossMock = mock(OSS.class, delegatesTo(oss));
+
+  private final Random random = new Random(1);
+  private final Path tmpDir = Files.createTempDirectory("oss-file-io-test-");

Review comment:
       Nit: Doesn't need terminal `-`.

##########
File path: aliyun/src/test/java/org/apache/iceberg/aliyun/oss/TestOSSOutputStream.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.aliyun.oss;
+
+import com.aliyun.oss.OSS;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Random;
+import java.util.UUID;
+import org.apache.commons.io.IOUtils;
+import org.apache.iceberg.aliyun.AliyunProperties;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.mockito.AdditionalAnswers.delegatesTo;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.reset;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+public class TestOSSOutputStream extends AliyunOSSTestBase {
+  private static final Logger LOG = LoggerFactory.getLogger(TestOSSOutputStream.class);
+
+  private final OSS oss = oss().get();
+  private final OSS ossMock = mock(OSS.class, delegatesTo(oss));
+
+  private final Random random = new Random(1);

Review comment:
       As this is a class variable, could you please put it up as a static final variable? It seems odd to me seeing it not as one.
   
   Or potentially using a thread local random function as a private static final value instead would be the most performant as random vs thread local random can be somewhat noticeable in added test time over time across whole repos.

##########
File path: aliyun/src/test/java/org/apache/iceberg/aliyun/oss/TestOSSURI.java
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.aliyun.oss;
+
+import org.apache.iceberg.AssertHelpers;
+import org.apache.iceberg.exceptions.ValidationException;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.junit.Assert;
+import org.junit.Test;
+
+import static com.aliyun.oss.internal.OSSUtils.OSS_RESOURCE_MANAGER;
+
+public class TestOSSURI {
+  @Test
+  public void testUrlParsing() {
+    String location = "oss://bucket/path/to/file";
+    OSSURI uri = new OSSURI(location);
+
+    Assert.assertEquals("bucket", uri.bucket());
+    Assert.assertEquals("path/to/file", uri.key());
+    Assert.assertEquals(location, uri.toString());
+  }
+
+  @Test
+  public void testEncodedString() {
+    String location = "oss://bucket/path%20to%20file";
+    OSSURI uri = new OSSURI(location);
+
+    Assert.assertEquals("bucket", uri.bucket());
+    Assert.assertEquals("path%20to%20file", uri.key());
+    Assert.assertEquals(location, uri.toString());
+  }
+
+  @Test
+  public void invalidOBucket() {

Review comment:
       Possible typo? Maybe it's a naming convention I'm not familiar with, in which case ignore / close this comment 😅 

##########
File path: aliyun/src/test/java/org/apache/iceberg/aliyun/oss/TestOSSOutputStream.java
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.aliyun.oss;
+
+import com.aliyun.oss.OSS;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Random;
+import java.util.UUID;
+import org.apache.commons.io.IOUtils;
+import org.apache.iceberg.aliyun.AliyunProperties;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.mockito.AdditionalAnswers.delegatesTo;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.reset;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+
+public class TestOSSOutputStream extends AliyunOSSTestBase {
+  private static final Logger LOG = LoggerFactory.getLogger(TestOSSOutputStream.class);
+
+  private final OSS oss = oss().get();
+  private final OSS ossMock = mock(OSS.class, delegatesTo(oss));
+
+  private final Random random = new Random(1);
+  private final Path tmpDir = Files.createTempDirectory("oss-file-io-test-");
+
+  private AliyunProperties props = new AliyunProperties(ImmutableMap.of(
+      AliyunProperties.OSS_STAGING_DIRECTORY, tmpDir.toString()
+  ));
+
+  public TestOSSOutputStream() throws IOException {
+  }
+
+  @Test
+  public void testWrite() throws IOException {
+    OSSURI uri = randomURI();
+
+    for (int i = 0; i < 2; i++) {
+      boolean arrayWrite = i % 2 == 0;
+      // Write small file.
+      writeAndVerify(ossMock, uri, data256(), arrayWrite);
+      verify(ossMock, times(1)).putObject(any());
+      reset(ossMock);
+
+      // Write large file.
+      writeAndVerify(ossMock, uri, randomData(32 * 1024 * 1024), arrayWrite);
+      verify(ossMock, times(1)).putObject(any());
+      reset(ossMock);
+    }
+  }
+
+  private void writeAndVerify(OSS mock, OSSURI uri, byte[] data, boolean arrayWrite)
+      throws IOException {
+    LOG.info("Write and verify for arguments uri: {}, data length: {}, arrayWrite: {}",
+            uri, data.length, arrayWrite);
+
+    try (OSSOutputStream out = new OSSOutputStream(mock, uri, props)) {
+      if (arrayWrite) {
+        out.write(data);
+        Assert.assertEquals(data.length, out.getPos());
+      } else {
+        for (int i = 0; i < data.length; i++) {
+          out.write(data[i]);
+          Assert.assertEquals(i + 1, out.getPos());
+        }
+      }
+    }
+
+    Assert.assertTrue(oss.doesObjectExist(uri.bucket(), uri.key()));
+    Assert.assertEquals(
+        oss.getObject(uri.bucket(), uri.key()).getObjectMetadata().getContentLength(),
+        data.length);
+    byte[] actual = new byte[data.length];
+    IOUtils.readFully(oss.getObject(uri.bucket(), uri.key()).getObjectContent(), actual);
+    Assert.assertArrayEquals(data, actual);

Review comment:
       Nit: Can you possibly add an assertion message to this test?

##########
File path: aliyun/src/main/java/org/apache/iceberg/aliyun/AliyunProperties.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.aliyun;
+
+import java.io.Serializable;
+import java.util.Map;
+import org.apache.iceberg.util.PropertyUtil;
+
+public class AliyunProperties implements Serializable {
+  /**
+   * Location to put staging files for uploading to OSS, default to temp directory set in java.io.tmpdir.

Review comment:
       Nit: should be `defaults`.
   
   Might just be easier to say `...for uploading to OSS, defaults to the directory value of java.io.tmpdir`. Leave that up to you 🙂 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org