You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/02/11 05:14:08 UTC
[GitHub] [ozone] guihecheng opened a new pull request #3076: HDDS-6295. EC: Fix unaligned stripe write failure due to length overflow.
guihecheng opened a new pull request #3076:
URL: https://github.com/apache/ozone/pull/3076
Co-authored-by: Uma Maheswara Rao G <um...@cloudera.com>
## What changes were proposed in this pull request?
Fix unaligned stripe write failure due to length overflow and
eliminate the unnecessary variable `currentBlockGroupLen`.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6295
## How was this patch tested?
existing UT in CI: https://github.com/guihecheng/ozone/actions/runs/1827271876
manual test:
./bin/ozone sh bucket create vol1/bucket1 --layout=FILE_SYSTEM_OPTIMIZED --replication=rs-10-4-1024k --type EC
./bin/ozone sh key put /vol1/bucket1/hosts /etc/hosts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on a change in pull request #3076: HDDS-6295. EC: Fix unaligned stripe write failure due to length overflow.
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #3076:
URL: https://github.com/apache/ozone/pull/3076#discussion_r804383087
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java
##########
@@ -587,8 +578,17 @@ public static void padBufferToLimit(ByteBuffer buf, int limit) {
buf.position(limit);
}
- private boolean isPartialStripe() {
- return currentBlockGroupLen % (numDataBlks * ecChunkSize) > 0;
+ private boolean isPartialStripe(int stripeSize) {
+ return stripeSize > 0 && stripeSize < numDataBlks * ecChunkSize;
+ }
+
+ private int getCurrentStripeSize() {
Review comment:
Could you please rename this to getCurrentDataStripeSize?
Generally we refer stripe size as combination of data+parity. We we need only data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on a change in pull request #3076: HDDS-6295. EC: Fix unaligned stripe write failure due to length overflow.
Posted by GitBox <gi...@apache.org>.
guihecheng commented on a change in pull request #3076:
URL: https://github.com/apache/ozone/pull/3076#discussion_r804388088
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java
##########
@@ -587,8 +578,17 @@ public static void padBufferToLimit(ByteBuffer buf, int limit) {
buf.position(limit);
}
- private boolean isPartialStripe() {
- return currentBlockGroupLen % (numDataBlks * ecChunkSize) > 0;
+ private boolean isPartialStripe(int stripeSize) {
+ return stripeSize > 0 && stripeSize < numDataBlks * ecChunkSize;
+ }
+
+ private int getCurrentStripeSize() {
Review comment:
Sure, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on pull request #3076: HDDS-6295. EC: Fix unaligned stripe write failure due to length overflow.
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #3076:
URL: https://github.com/apache/ozone/pull/3076#issuecomment-1035915881
I really want a test for this.
Below is the test code reproduces it. Please include this and update. Thanks a lot for your help @guihecheng
```
@Test
public void test10D4PConfigWithPartialStripe()
throws IOException {
conf.set(OzoneConfigKeys.OZONE_SCM_BLOCK_SIZE,
OzoneConfigKeys.OZONE_SCM_BLOCK_SIZE_DEFAULT);
int dataBlks = 10;
int parityBlk = 4;
store.createVolume(volumeName);
OzoneVolume volume = store.getVolume(volumeName);
volume.createBucket(bucketName);
OzoneBucket bucket = volume.getBucket(bucketName);
try (OzoneOutputStream out = bucket.createKey(keyName, chunkSize,
new ECReplicationConfig(dataBlks, parityBlk,
ECReplicationConfig.EcCodec.RS, chunkSize), new HashMap<>())) {
out.write(new byte[1023]);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao merged pull request #3076: HDDS-6295. EC: Fix unaligned stripe write failure due to length overflow.
Posted by GitBox <gi...@apache.org>.
umamaheswararao merged pull request #3076:
URL: https://github.com/apache/ozone/pull/3076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org