You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Sammi Chen (Jira)" <ji...@apache.org> on 2021/08/06 10:09:00 UTC
[jira] [Created] (HDDS-5548) Keep downloaded .gz.tar container file
for debug purpose
Sammi Chen created HDDS-5548:
--------------------------------
Summary: Keep downloaded .gz.tar container file for debug purpose
Key: HDDS-5548
URL: https://issues.apache.org/jira/browse/HDDS-5548
Project: Apache Ozone
Issue Type: Improvement
Reporter: Sammi Chen
Assignee: Sammi Chen
There are a lot of container import failure LOGs in production, such as,
2021-08-03 21:48:12,311 [ContainerReplicationThread-9] INFO org.apache.hadoop.ozone.container.replication.DownloadAndImportReplicator: Starting replication of container 66315 from [4e613295-6d55-4bf9-bdc9-1668fd24741c{ip: 11.61.44.244, host: 11.61.44.244, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9858, RATIS_SERVER=9858, STANDALONE=9859], networkLocation: /rack582702, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}, 7694e208-c887-4d8e-b249-28a176b4d7b7{ip: 11.61.45.38, host: 11.61.45.38, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9858, RATIS_SERVER=9858, STANDALONE=9859], networkLocation: /rack582788, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}]
2021-08-03 21:48:17,462 [grpc-default-executor-12557] INFO org.apache.hadoop.ozone.container.replication.GrpcReplicationClient: Container 66315 is downloaded to /data/ozoneadmin/ozoneenv/ozone-temp/container-66315.tar.gz
2021-08-03 21:48:17,462 [ContainerReplicationThread-9] INFO org.apache.hadoop.ozone.container.replication.DownloadAndImportReplicator: Container 66315 is downloaded with size 6154503, starting to import.
2021-08-03 21:48:17,582 [ContainerReplicationThread-9] ERROR org.apache.hadoop.ozone.container.replication.DownloadAndImportReplicator: Container 66315 replication was unsuccessful.
java.io.IOException: Container descriptor is missing from the container archive.
at org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:190)
at org.apache.hadoop.ozone.container.replication.DownloadAndImportReplicator.importContainer(DownloadAndImportReplicator.java:76)
at org.apache.hadoop.ozone.container.replication.DownloadAndImportReplicator.replicate(DownloadAndImportReplicator.java:125)
at org.apache.hadoop.ozone.container.replication.MeasuredReplicator.replicate(MeasuredReplicator.java:69)
at org.apache.hadoop.ozone.container.replication.ReplicationSupervisor$TaskRunner.run(ReplicationSupervisor.java:139)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-08-03 21:48:17,582 [ContainerReplicationThread-9] ERROR org.apache.hadoop.ozone.container.replication.ReplicationSupervisor: Container 66315 can't be downloaded from any of the datanodes.
In the above case, 66315 container on the source datanode actually has the Container descriptor on disk. So what's the root cause of this error is in doubt.
This task is to keep the downloaded tar file for investigation purpose at the cost of storage space.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org