You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/10/06 16:37:41 UTC
[GitHub] [ozone] sodonnel opened a new pull request #2717: HDDS-5832. EC: ECKeyOutputStream persists blocks in random order
sodonnel opened a new pull request #2717:
URL: https://github.com/apache/ozone/pull/2717
## What changes were proposed in this pull request?
When the blocks for an ECKeyOutputStream are persisted in OM, they are in a random order, which means the data cannot be read back successfully later.
This is because they are stored into a HashMap which does not iterate in insertion order. Instead we need to use a LinkedHashMap.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5832
## How was this patch tested?
New test to reproduce the issue before the fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on a change in pull request #2717: HDDS-5832. EC: ECKeyOutputStream persists blocks in random order
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2717:
URL: https://github.com/apache/ozone/pull/2717#discussion_r723800435
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java
##########
@@ -195,6 +195,7 @@ public void write(byte[] b, int off, int len) throws IOException {
currentWriterChunkLenToWrite,
currentChunkBufferLen + currentWriterChunkLenToWrite == ecChunkSize);
checkAndWriteParityCells(pos);
+ off += currentWriterChunkLenToWrite;
Review comment:
Now this change committed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #2717: HDDS-5832. EC: ECKeyOutputStream persists blocks in random order
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #2717:
URL: https://github.com/apache/ozone/pull/2717#discussion_r723480666
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java
##########
@@ -195,6 +195,7 @@ public void write(byte[] b, int off, int len) throws IOException {
currentWriterChunkLenToWrite,
currentChunkBufferLen + currentWriterChunkLenToWrite == ecChunkSize);
checkAndWriteParityCells(pos);
+ off += currentWriterChunkLenToWrite;
Review comment:
Note this duplicates the change in #2714 so this change will be removed from here when it is committed (before this one)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a change in pull request #2717: HDDS-5832. EC: ECKeyOutputStream persists blocks in random order
Posted by GitBox <gi...@apache.org>.
sodonnel commented on a change in pull request #2717:
URL: https://github.com/apache/ozone/pull/2717#discussion_r723480666
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java
##########
@@ -195,6 +195,7 @@ public void write(byte[] b, int off, int len) throws IOException {
currentWriterChunkLenToWrite,
currentChunkBufferLen + currentWriterChunkLenToWrite == ecChunkSize);
checkAndWriteParityCells(pos);
+ off += currentWriterChunkLenToWrite;
Review comment:
Note this duplicates the change in #2714 so this change will be removed from here with it is committed (before this one)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on a change in pull request #2717: HDDS-5832. EC: ECKeyOutputStream persists blocks in random order
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on a change in pull request #2717:
URL: https://github.com/apache/ozone/pull/2717#discussion_r723800435
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java
##########
@@ -195,6 +195,7 @@ public void write(byte[] b, int off, int len) throws IOException {
currentWriterChunkLenToWrite,
currentChunkBufferLen + currentWriterChunkLenToWrite == ecChunkSize);
checkAndWriteParityCells(pos);
+ off += currentWriterChunkLenToWrite;
Review comment:
Now this change committed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao merged pull request #2717: HDDS-5832. EC: ECKeyOutputStream persists blocks in random order
Posted by GitBox <gi...@apache.org>.
umamaheswararao merged pull request #2717:
URL: https://github.com/apache/ozone/pull/2717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] umamaheswararao commented on pull request #2717: HDDS-5832. EC: ECKeyOutputStream persists blocks in random order
Posted by GitBox <gi...@apache.org>.
umamaheswararao commented on pull request #2717:
URL: https://github.com/apache/ozone/pull/2717#issuecomment-938227009
I have just merged this to branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org