You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/09/09 08:20:00 UTC
[GitHub] [ozone] cchenax opened a new pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
cchenax opened a new pull request #2628:
URL: https://github.com/apache/ozone/pull/2628
## What changes were proposed in this pull request?
when concurrent write,for example,the chunkinfo{offset is 8192000,chunk size is 4096000} write in chunkfile,but the chunkinfo{offset is 4096000,chunk size is 4096000} is not write in chunkfile,so when this chunkinfo write in chunkfile will print warning log datanode duplicate write,but this chunkinfo was not wrote before.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5732
## How was this patch tested?
ci
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on a change in pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
Posted by GitBox <gi...@apache.org>.
kaijchen commented on a change in pull request #2628:
URL: https://github.com/apache/ozone/pull/2628#discussion_r705342883
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/ChunkUtils.java
##########
@@ -256,12 +257,21 @@ public static boolean validateChunkForOverwrite(File chunkFile,
public static boolean isOverWriteRequested(File chunkFile, ChunkInfo
chunkInfo) {
- if (!chunkFile.exists()) {
+ long offset = chunkInfo.getOffset();
+ try {
+ FileInputStream inputStream = new FileInputStream(chunkFile);
+ inputStream.getChannel().position(offset);
Review comment:
Can we use `chunkFile.length()` instead of seek and read?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on a change in pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
Posted by GitBox <gi...@apache.org>.
kaijchen commented on a change in pull request #2628:
URL: https://github.com/apache/ozone/pull/2628#discussion_r705865472
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/ChunkUtils.java
##########
@@ -256,12 +257,21 @@ public static boolean validateChunkForOverwrite(File chunkFile,
public static boolean isOverWriteRequested(File chunkFile, ChunkInfo
chunkInfo) {
- if (!chunkFile.exists()) {
+ long offset = chunkInfo.getOffset();
+ try {
+ FileInputStream inputStream = new FileInputStream(chunkFile);
+ inputStream.getChannel().position(offset);
Review comment:
If my understanding is correct, you are trying to solve sparse file "overwrite" problem.
But this code still prints true.
```java
public static void main(String[] args) {
File f = new File("/tmp/hello");
try (FileOutputStream out = new FileOutputStream(f);
FileInputStream in = new FileInputStream(f)) {
out.getChannel().position(8 << 10); // 8KB
out.write("hello, world".getBytes(StandardCharsets.UTF_8));
System.out.println(f.length()); // prints 8204 (= 8192 + 12)
in.getChannel().position(4 << 10); // 4KB
if (in.read() == -1) { // read() = 0
System.out.println(false);
} else {
System.out.println(true); // prints true
}
} catch (IOException e) {
System.out.println(e);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on a change in pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
Posted by GitBox <gi...@apache.org>.
kaijchen commented on a change in pull request #2628:
URL: https://github.com/apache/ozone/pull/2628#discussion_r706010403
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/ChunkUtils.java
##########
@@ -256,12 +257,21 @@ public static boolean validateChunkForOverwrite(File chunkFile,
public static boolean isOverWriteRequested(File chunkFile, ChunkInfo
chunkInfo) {
- if (!chunkFile.exists()) {
+ long offset = chunkInfo.getOffset();
+ try {
+ FileInputStream inputStream = new FileInputStream(chunkFile);
+ inputStream.getChannel().position(offset);
Review comment:
Can you give an example of inputs, when your code and the original code will return different results?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #2628:
URL: https://github.com/apache/ozone/pull/2628#issuecomment-918853304
Thanks @cchenax for the patch. To check the overwrite flag , seek and trying to read something may not be a viable option. With sparse files, read should return 0's not EOF. Do we still see concurrent writes for the same block?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bshashikant commented on pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #2628:
URL: https://github.com/apache/ozone/pull/2628#issuecomment-932168093
@cchenax , any update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi edited a comment on pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
Posted by GitBox <gi...@apache.org>.
ChenSammi edited a comment on pull request #2628:
URL: https://github.com/apache/ozone/pull/2628#issuecomment-938461849
Hi @bshashikant , the issue is a false alert. I will close the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] cchenax commented on a change in pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
Posted by GitBox <gi...@apache.org>.
cchenax commented on a change in pull request #2628:
URL: https://github.com/apache/ozone/pull/2628#discussion_r705846254
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/ChunkUtils.java
##########
@@ -256,12 +257,21 @@ public static boolean validateChunkForOverwrite(File chunkFile,
public static boolean isOverWriteRequested(File chunkFile, ChunkInfo
chunkInfo) {
- if (!chunkFile.exists()) {
+ long offset = chunkInfo.getOffset();
+ try {
+ FileInputStream inputStream = new FileInputStream(chunkFile);
+ inputStream.getChannel().position(offset);
Review comment:
chunkfile.length depend on the offset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] cchenax commented on a change in pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
Posted by GitBox <gi...@apache.org>.
cchenax commented on a change in pull request #2628:
URL: https://github.com/apache/ozone/pull/2628#discussion_r705967817
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/ChunkUtils.java
##########
@@ -256,12 +257,21 @@ public static boolean validateChunkForOverwrite(File chunkFile,
public static boolean isOverWriteRequested(File chunkFile, ChunkInfo
chunkInfo) {
- if (!chunkFile.exists()) {
+ long offset = chunkInfo.getOffset();
+ try {
+ FileInputStream inputStream = new FileInputStream(chunkFile);
+ inputStream.getChannel().position(offset);
Review comment:
Ozone write the contents of the buffer from the current position to the limit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi closed pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
Posted by GitBox <gi...@apache.org>.
ChenSammi closed pull request #2628:
URL: https://github.com/apache/ozone/pull/2628
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #2628: HDDS-5732 datanode duplicate write when concurrent write
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on pull request #2628:
URL: https://github.com/apache/ozone/pull/2628#issuecomment-938461849
The issue is a false alert. Close the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org