You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2024/07/13 07:25:58 UTC
[PR] HDDS-11171. [DN] Add EC Block Recover Audit Log. [ozone]
slfan1989 opened a new pull request, #6936:
URL: https://github.com/apache/ozone/pull/6936
## What changes were proposed in this pull request?
Provide a one-liner summary of the changes in the PR **Title** field above.
It should be in the form of `HDDS-1234. Short summary of the change`.
Please describe your PR in detail:
* What changes are proposed in the PR? and Why? It would be better if it is written from third person's
perspective not just for the reviewer.
* Provide as much context and rationale for the pull request as possible. It could be copy-paste from
the Jira's description if the jira is well defined.
* If it is complex code, describe the approach used to solve the issue. If possible attach design doc,
issue investigation, github discussion, etc.
Examples of well-written pull requests:
* [#3980](https://github.com/apache/ozone/pull/3980)
* [#5265](https://github.com/apache/ozone/pull/5265)
* [#4701](https://github.com/apache/ozone/pull/4701)
* [#5283](https://github.com/apache/ozone/pull/5283)
* [#5300](https://github.com/apache/ozone/pull/5300)
## What is the link to the Apache JIRA
JIRA: HDDS-11171. [DN] Add EC Block Recover Audit Log.
## How was this patch tested?
(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11171. [DN] Add EC Block Recover Audit Log. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6936:
URL: https://github.com/apache/ozone/pull/6936#issuecomment-2227739713
Thanks @slfan1989 for working on this.
> reconstruction process may either succeed or fail. To swiftly grasp the outcome of EC block reconstruction
The coordinator datanode already logs status/result of EC reconstruction, search for `reconstructECContainersCommand` (see examples below).
> implement an auditing feature dedicated to EC reconstruction logs
Datanode currently only writes audit log for container state machine commands in `HddsDispatcher`. Auditing commands from SCM might be useful, but:
- I don't think it should be specific to EC recovery. It would be better added in a generic way in `CommandDispatcher`.
- Need to consider both sync and async command handlers.
----
Start:
```
2024-07-13 14:27:56,098 [f3ffd8a7-4b5c-47f5-98af-404eba82b826-ContainerReplicationThread-0] INFO reconstruction.ECReconstructionCoordinatorTask: IN_PROGRESS reconstructECContainersCommand: containerID=1, replication=rs-3-2-1024k, missingIndexes=, sources={1=e237cf49-df14-416e-ab15-8469b59d04c8(ozone-datanode-1.ozone_default/172.18.0.9), 3=8ee9d1ce-75a9-493f-905d-b1981f13aa00(ozone-datanode-3.ozone_default/172.18.0.10), 5=033eca1e-07e5-4893-b5a1-68dcccbe8c9c(ozone-datanode-2.ozone_default/172.18.0.8)}, targets={2=f3ffd8a7-4b5c-47f5-98af-404eba82b826(ozone-datanode-4.ozone_default/172.18.0.6), 4=2c4209f5-c826-4ed2-9658-b23961f074d6(ozone-datanode-5.ozone_default/172.18.0.11)}
```
Success:
```
2024-07-13 14:28:04,066 [f3ffd8a7-4b5c-47f5-98af-404eba82b826-ContainerReplicationThread-0] INFO reconstruction.ECReconstructionCoordinatorTask: DONE reconstructECContainersCommand: containerID=1, replication=rs-3-2-1024k, missingIndexes=, sources={1=e237cf49-df14-416e-ab15-8469b59d04c8(ozone-datanode-1.ozone_default/172.18.0.9), 3=8ee9d1ce-75a9-493f-905d-b1981f13aa00(ozone-datanode-3.ozone_default/172.18.0.10), 5=033eca1e-07e5-4893-b5a1-68dcccbe8c9c(ozone-datanode-2.ozone_default/172.18.0.8)}, targets={2=f3ffd8a7-4b5c-47f5-98af-404eba82b826(ozone-datanode-4.ozone_default/172.18.0.6), 4=2c4209f5-c826-4ed2-9658-b23961f074d6(ozone-datanode-5.ozone_default/172.18.0.11)} in 7968 ms
```
Failure:
```
2024-03-24 12:17:19,479 [nullContainerReplicationThread-1] WARN reconstruction.ECReconstructionCoordinatorTask: FAILED reconstructECContainersCommand: containerID=5, replication=rs-3-2-1024k, missingIndexes=[1], sources={2=165ad6c9-23bb-4b60-a209-0f37fb43c778(ozonesecure_datanode_1.ozonesecure_default/172.19.0.8), 3=33dddf06-b56a-4475-8523-ad54a589c1b3(ozonesecure_datanode_2.ozonesecure_default/172.19.0.10), 4=3575b633-fb5a-4c41-9ece-c2f97994544f(ozonesecure_datanode_4.ozonesecure_default/172.19.0.13), 5=b56dc487-1437-4c61-a277-f7318a61d93d(ozonesecure_datanode_3.ozonesecure_default/172.19.0.11)}, targets={1=ea3e9265-f4d3-4424-b4b2-d14a5cf5c9a7(ozonesecure_datanode_5.ozonesecure_default/172.19.0.13)} after 1090 ms
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: ContainerID 5 does not exist
at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:653)
...
at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.listBlock(ContainerProtocolCalls.java:134)
at org.apache.hadoop.ozone.container.ec.reconstruction.ECContainerOperationClient.listBlock(ECContainerOperationClient.java:96)
at org.apache.hadoop.ozone.container.ec.reconstruction.ECReconstructionCoordinator.getBlockDataMap(ECReconstructionCoordinator.java:505)
at org.apache.hadoop.ozone.container.ec.reconstruction.ECReconstructionCoordinator.reconstructECContainerGroup(ECReconstructionCoordinator.java:153)
at org.apache.hadoop.ozone.container.ec.reconstruction.ECReconstructionCoordinatorTask.runTask(ECReconstructionCoordinatorTask.java:68)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11171. [DN] Add EC Block Recover Audit Log. [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #6936:
URL: https://github.com/apache/ozone/pull/6936#issuecomment-2229760429
Hi @slfan1989, I agree more diagnostic info would be helpful. As Attila said, the datanode audit log is for requests it receives from clients. That is why it has `user` and `ip` fields which are null in your example. It currently does not have logging for intra-datanode operations. Ozone already has a feature to "audit" changes to containers which was added in HDDS-8062 and might be a better fit for this type of thing. The file `dn-container.log` currently only logs a container as recovered on [success](https://github.com/apache/ozone/blob/930d3f0552b86d6605625cabe5e8961652d14fa2/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java#L1051), but adding logging for failure scenarios would be useful too. There is logging for failed container scans there as well.
Also, I see the example success log is at the block level, but EC reconstruction happens at the container level. Having one success or failure log for every block would probably be too verbose. If we can just log once per container on success or failure that would be more concise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11171. [DN] Add EC Block Recover Audit Log. [ozone]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #6936:
URL: https://github.com/apache/ozone/pull/6936#issuecomment-2227717942
@sodonnel Can you help review this pr? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11171. [DN] Add EC Block Recover Audit Log. [ozone]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #6936:
URL: https://github.com/apache/ozone/pull/6936#issuecomment-2229917289
@adoroszlai @errose28 Thank you very much for your response! it has been very helpful and insightful. I understand that adding this operation to the audit log may not be appropriate. The primary reason for logging this is to address a practical issue: some EC blocks currently fail to recover successfully due to variations in their BlockGroupLength. This results in repeated repair attempts on different DataNodes in the cluster. My objective with the audit log is to identify the blocks that failed reconstruction. Subsequently, by using containerId and locationId, I aim to locate the corresponding files. Given the potentially limited number of affected files, my current approach involves retrieving these files using "get" operations and then restoring them using "put" operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org