You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/10/22 02:32:18 UTC
[GitHub] [ozone] guohao-rosicky opened a new pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
guohao-rosicky opened a new pull request #2760:
URL: https://github.com/apache/ozone/pull/2760
OzoneBucket and RpcClient not support multipart upload with Stream.
Add the createMultipartStreamKey method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#issuecomment-973655576
> @guohao-rosicky , TestOzoneRpcClientAbstract.testMultiPartUploadWithStream failed again in the last build. Please take a look.
Thanks @szetszwo , please TBR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#issuecomment-949488809
jira url: https://issues.apache.org/jira/browse/HDDS-5879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#issuecomment-949491026
Thanks @captainzmc , has been modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#discussion_r734702585
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
##########
@@ -1117,6 +1117,69 @@ public OzoneOutputStream createMultipartKey(String volumeName,
}
}
+ @Override
+ public OzoneDataStreamOutput createMultipartStreamKey(
+ String volumeName,
+ String bucketName,
+ String keyName,
+ long size,
+ int partNumber,
+ String uploadID,
+ ReplicationConfig replicationConfig)
+ throws IOException {
+ verifyVolumeName(volumeName);
+ verifyBucketName(bucketName);
+ if(checkKeyNameEnabled) {
+ HddsClientUtils.verifyKeyName(keyName);
+ }
+ HddsClientUtils.checkNotNull(keyName, uploadID, replicationConfig);
+ Preconditions.checkArgument(partNumber > 0 && partNumber <=10000, "Part " +
+ "number should be greater than zero and less than or equal to 10000");
+ Preconditions.checkArgument(size >=0, "size should be greater than or " +
+ "equal to zero");
+ String requestId = UUID.randomUUID().toString();
+
+ OmKeyArgs keyArgs = new OmKeyArgs.Builder()
+ .setVolumeName(volumeName)
+ .setBucketName(bucketName)
+ .setKeyName(keyName)
+ .setDataSize(size)
+ .setIsMultipartKey(true)
+ .setMultipartUploadID(uploadID)
+ .setMultipartUploadPartNumber(partNumber)
+ .setAcls(getAclList())
+ .build();
+
+ OpenKeySession openKey = ozoneManagerClient.openKey(keyArgs);
+
+ KeyDataStreamOutput keyOutputStream =
+ new KeyDataStreamOutput.Builder()
+ .setHandler(openKey)
+ .setXceiverClientManager(xceiverClientManager)
+ .setOmClient(ozoneManagerClient)
+ .setRequestID(requestId)
+ .setReplicationConfig(replicationConfig)
+ .setMultipartNumber(partNumber)
+ .setMultipartUploadID(uploadID)
+ .setIsMultipartKey(true)
+ .enableUnsafeByteBufferConversion(unsafeByteBufferConversion)
+ .setConfig(clientConfig)
+ .build();
+ keyOutputStream
+ .addPreallocateBlocks(
+ openKey.getKeyInfo().getLatestVersionLocations(),
+ openKey.getOpenVersion());
+
+ FileEncryptionInfo feInfo = openKey.getKeyInfo().getFileEncryptionInfo();
+ if (feInfo != null) {
Review comment:
Can we create a Jira for this TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on a change in pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on a change in pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#discussion_r735049487
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
##########
@@ -1117,6 +1117,69 @@ public OzoneOutputStream createMultipartKey(String volumeName,
}
}
+ @Override
+ public OzoneDataStreamOutput createMultipartStreamKey(
+ String volumeName,
+ String bucketName,
+ String keyName,
+ long size,
+ int partNumber,
+ String uploadID,
+ ReplicationConfig replicationConfig)
+ throws IOException {
+ verifyVolumeName(volumeName);
+ verifyBucketName(bucketName);
+ if(checkKeyNameEnabled) {
+ HddsClientUtils.verifyKeyName(keyName);
+ }
+ HddsClientUtils.checkNotNull(keyName, uploadID, replicationConfig);
+ Preconditions.checkArgument(partNumber > 0 && partNumber <=10000, "Part " +
+ "number should be greater than zero and less than or equal to 10000");
+ Preconditions.checkArgument(size >=0, "size should be greater than or " +
+ "equal to zero");
+ String requestId = UUID.randomUUID().toString();
+
+ OmKeyArgs keyArgs = new OmKeyArgs.Builder()
+ .setVolumeName(volumeName)
+ .setBucketName(bucketName)
+ .setKeyName(keyName)
+ .setDataSize(size)
+ .setIsMultipartKey(true)
+ .setMultipartUploadID(uploadID)
+ .setMultipartUploadPartNumber(partNumber)
+ .setAcls(getAclList())
+ .build();
+
+ OpenKeySession openKey = ozoneManagerClient.openKey(keyArgs);
+
+ KeyDataStreamOutput keyOutputStream =
+ new KeyDataStreamOutput.Builder()
+ .setHandler(openKey)
+ .setXceiverClientManager(xceiverClientManager)
+ .setOmClient(ozoneManagerClient)
+ .setRequestID(requestId)
+ .setReplicationConfig(replicationConfig)
+ .setMultipartNumber(partNumber)
+ .setMultipartUploadID(uploadID)
+ .setIsMultipartKey(true)
+ .enableUnsafeByteBufferConversion(unsafeByteBufferConversion)
+ .setConfig(clientConfig)
+ .build();
+ keyOutputStream
+ .addPreallocateBlocks(
+ openKey.getKeyInfo().getLatestVersionLocations(),
+ openKey.getOpenVersion());
+
+ FileEncryptionInfo feInfo = openKey.getKeyInfo().getFileEncryptionInfo();
+ if (feInfo != null) {
Review comment:
Thanks @bharatviswa504 , new jira url: https://issues.apache.org/jira/browse/HDDS-5892
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
szetszwo commented on pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#issuecomment-971429839
@guohao-rosicky , TestOzoneRpcClientAbstract.testMultiPartUploadWithStream failed again in the last build. Please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on a change in pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on a change in pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#discussion_r749307061
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java
##########
@@ -730,6 +730,23 @@ public OzoneOutputStream createMultipartKey(String key, long size,
uploadID);
}
+ /**
+ * Create a part key for a multipart upload key.
+ * @param key
+ * @param size
+ * @param partNumber
+ * @param uploadID
+ * @param replicationConfig
+ * @return OzoneDataStreamOutput
+ * @throws IOException
+ */
+ public OzoneDataStreamOutput createMultipartStreamKey(String key,
+ long size, int partNumber, String uploadID,
+ ReplicationConfig replicationConfig) throws IOException {
Review comment:
@szetszwo
I noticed it. Thanks for reminding me. I have revised it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo merged pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
szetszwo merged pull request #2760:
URL: https://github.com/apache/ozone/pull/2760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guohao-rosicky commented on pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
guohao-rosicky commented on pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#issuecomment-950499246
Thanks @captainzmc , already added UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#issuecomment-949281846
Thanks @guohao-rosicky for your contribution. Could you update the URL of jira in comment? Also, the current code check style still has some problems, and it is better to add some UT to test the new interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on a change in pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#discussion_r738073359
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java
##########
@@ -730,6 +730,23 @@ public OzoneOutputStream createMultipartKey(String key, long size,
uploadID);
}
+ /**
+ * Create a part key for a multipart upload key.
+ * @param key
+ * @param size
+ * @param partNumber
+ * @param uploadID
+ * @param replicationConfig
+ * @return OzoneDataStreamOutput
+ * @throws IOException
+ */
+ public OzoneDataStreamOutput createMultipartStreamKey(String key,
+ long size, int partNumber, String uploadID,
+ ReplicationConfig replicationConfig) throws IOException {
Review comment:
Why we pass replicationConfig but not use openKey.getKeyInfo().getReplicationConfig()?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on a change in pull request #2760: HDDS-5879. [Ozone-Streaming] OzoneBucket add the createMultipartStreamKey method
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #2760:
URL: https://github.com/apache/ozone/pull/2760#discussion_r750114870
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
##########
@@ -1133,6 +1133,68 @@ public OzoneOutputStream createMultipartKey(String volumeName,
}
}
+ @Override
+ public OzoneDataStreamOutput createMultipartStreamKey(
+ String volumeName,
+ String bucketName,
+ String keyName,
+ long size,
+ int partNumber,
+ String uploadID)
+ throws IOException {
+ verifyVolumeName(volumeName);
+ verifyBucketName(bucketName);
+ if (checkKeyNameEnabled) {
+ HddsClientUtils.verifyKeyName(keyName);
+ }
+ HddsClientUtils.checkNotNull(keyName, uploadID);
+ Preconditions.checkArgument(partNumber > 0 && partNumber <= 10000, "Part " +
+ "number should be greater than zero and less than or equal to 10000");
+ Preconditions.checkArgument(size >= 0, "size should be greater than or " +
+ "equal to zero");
+ String requestId = UUID.randomUUID().toString();
+
+ OmKeyArgs keyArgs = new OmKeyArgs.Builder()
+ .setVolumeName(volumeName)
+ .setBucketName(bucketName)
+ .setKeyName(keyName)
+ .setDataSize(size)
+ .setIsMultipartKey(true)
+ .setMultipartUploadID(uploadID)
+ .setMultipartUploadPartNumber(partNumber)
+ .setAcls(getAclList())
+ .build();
+
+ OpenKeySession openKey = ozoneManagerClient.openKey(keyArgs);
+
+ KeyDataStreamOutput keyOutputStream =
+ new KeyDataStreamOutput.Builder()
+ .setHandler(openKey)
+ .setXceiverClientManager(xceiverClientManager)
+ .setOmClient(ozoneManagerClient)
+ .setRequestID(requestId)
+ .setReplicationConfig(openKey.getKeyInfo().getReplicationConfig())
+ .setMultipartNumber(partNumber)
+ .setMultipartUploadID(uploadID)
+ .setIsMultipartKey(true)
+ .enableUnsafeByteBufferConversion(unsafeByteBufferConversion)
+ .setConfig(clientConfig)
+ .build();
+ keyOutputStream
+ .addPreallocateBlocks(
+ openKey.getKeyInfo().getLatestVersionLocations(),
+ openKey.getOpenVersion());
+
+ FileEncryptionInfo feInfo = openKey.getKeyInfo().getFileEncryptionInfo();
+ if (feInfo != null) {
+ // todo: need to support file encrypt,
+ // https://issues.apache.org/jira/browse/HDDS-5892
+ return null;
Review comment:
Let's throw a UnsupportedOperationException.
```
throw new UnsupportedOperationException(
"FileEncryptionInfo is not yet supported in createMultipartStreamKey");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org