You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/01/19 10:57:42 UTC
[GitHub] [ozone] Xushaohong opened a new pull request #3000: HDDS-6203. CleanUp incomplete gz files during Container move
Xushaohong opened a new pull request #3000:
URL: https://github.com/apache/ozone/pull/3000
## What changes were proposed in this pull request?
If the target temporary directory is small, the concurrent downloading threads will have a contest for space and leave incomplete files. These should be cleaned up.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6203
## How was this patch tested?
manual tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #3000: HDDS-6203. CleanUp incomplete gz files during Container move
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on pull request #3000:
URL: https://github.com/apache/ozone/pull/3000#issuecomment-1019398693
The last patch LGTM, +1.
Thanks @Xushaohong for the contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on a change in pull request #3000: HDDS-6203. CleanUp incomplete gz files during Container move
Posted by GitBox <gi...@apache.org>.
guihecheng commented on a change in pull request #3000:
URL: https://github.com/apache/ozone/pull/3000#discussion_r787706981
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/GrpcReplicationClient.java
##########
@@ -161,6 +161,9 @@ public void onNext(CopyContainerResponseProto chunk) {
try {
chunk.getData().writeTo(stream);
} catch (IOException e) {
+ LOG.error("Failed to write the stream buffer to {} for container {}",
+ outputPath, containerId, e);
+ deleteOutputOnFailure();
Review comment:
May be we could extract common error handling stuff into a new method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on a change in pull request #3000: HDDS-6203. CleanUp incomplete gz files during Container move
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #3000:
URL: https://github.com/apache/ozone/pull/3000#discussion_r788291861
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/GrpcReplicationClient.java
##########
@@ -161,6 +161,9 @@ public void onNext(CopyContainerResponseProto chunk) {
try {
chunk.getData().writeTo(stream);
} catch (IOException e) {
+ LOG.error("Failed to write the stream buffer to {} for container {}",
+ outputPath, containerId, e);
+ deleteOutputOnFailure();
Review comment:
stream.close(); should be called together with deleteOutputOnFailure();
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi edited a comment on pull request #3000: HDDS-6203. CleanUp incomplete gz files during Container move
Posted by GitBox <gi...@apache.org>.
ChenSammi edited a comment on pull request #3000:
URL: https://github.com/apache/ozone/pull/3000#issuecomment-1019398693
The last patch LGTM, +1.
Thanks @Xushaohong for the contribution, and @guihecheng @JacksonYao287 for the code review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #3000: HDDS-6203. CleanUp incomplete gz files during Container move
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #3000:
URL: https://github.com/apache/ozone/pull/3000#discussion_r787688128
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/GrpcReplicationClient.java
##########
@@ -161,6 +161,9 @@ public void onNext(CopyContainerResponseProto chunk) {
try {
chunk.getData().writeTo(stream);
} catch (IOException e) {
+ LOG.error("Failed to write the stream buffer to {} for container {}",
+ outputPath, containerId, e);
+ deleteOutputOnFailure();
Review comment:
```suggestion
onError(e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] guihecheng commented on a change in pull request #3000: HDDS-6203. CleanUp incomplete gz files during Container move
Posted by GitBox <gi...@apache.org>.
guihecheng commented on a change in pull request #3000:
URL: https://github.com/apache/ozone/pull/3000#discussion_r787705250
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/GrpcReplicationClient.java
##########
@@ -161,6 +161,9 @@ public void onNext(CopyContainerResponseProto chunk) {
try {
chunk.getData().writeTo(stream);
} catch (IOException e) {
+ LOG.error("Failed to write the stream buffer to {} for container {}",
+ outputPath, containerId, e);
+ deleteOutputOnFailure();
Review comment:
IMO, the original LOG msg is better, because by looking at the JIRA descriptions, actually the download via network is successful, the failure happens at sync the downloaded data into the on-disk file.
So we don't call onError() here which should be called on network transfer faliures, and it may not be a good style to call an overrided method in another(onError in onNext here).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JacksonYao287 commented on a change in pull request #3000: HDDS-6203. CleanUp incomplete gz files during Container move
Posted by GitBox <gi...@apache.org>.
JacksonYao287 commented on a change in pull request #3000:
URL: https://github.com/apache/ozone/pull/3000#discussion_r787717619
##########
File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/replication/GrpcReplicationClient.java
##########
@@ -161,6 +161,9 @@ public void onNext(CopyContainerResponseProto chunk) {
try {
chunk.getData().writeTo(stream);
} catch (IOException e) {
+ LOG.error("Failed to write the stream buffer to {} for container {}",
+ outputPath, containerId, e);
+ deleteOutputOnFailure();
Review comment:
> May be we could extract common error handling stuff into a new method.
looks good. i agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi merged pull request #3000: HDDS-6203. CleanUp incomplete gz files during Container move
Posted by GitBox <gi...@apache.org>.
ChenSammi merged pull request #3000:
URL: https://github.com/apache/ozone/pull/3000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org