You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/05/28 18:06:35 UTC
[GitHub] [hadoop-ozone] maobaolong opened a new pull request #983: HDDS-3584. Support write 2 replication
maobaolong opened a new pull request #983:
URL: https://github.com/apache/hadoop-ozone/pull/983
## What changes were proposed in this pull request?
https://docs.google.com/document/d/1YfN5IOckveREuwFWJvWJNHnySo2qSqsQ0e_sxE7XatY/edit#
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3584
## How was this patch tested?
- Normal tests.
```bash
# normal put
bin/ozone sh key put -r TWO /myvol/mybucket/NOTICE22.txt NOTICE.txt
# stop one of the datanode of pipeline's container of /myvol/mybucket/NOTICE22.txt
# wait until the datanode become stale, then to dead state, ensure the pipeline of it is close
# and the container's datanodes of the file are all alive after the replicationManager fire a replicate work.
bin/ozone sh key info /myvol/mybucket/NOTICE22.txt
{
"volumeName" : "myvol",
"bucketName" : "mybucket",
"name" : "NOTICE22.txt",
"dataSize" : 17540,
"creationTime" : "2020-05-28T13:56:00.047Z",
"modificationTime" : "2020-05-28T13:56:01.922Z",
"replicationType" : "RATIS",
"replicationFactor" : 2,
"ozoneKeyLocations" : [ {
"containerID" : 1,
"localID" : 104246421751595008,
"length" : 17540,
"offset" : 0
} ],
"metadata" : { },
"fileEncryptionInfo" : null
}
bin/ozone admin container info 1
Container id: 1
Pipeline id: 2c1dc5cc-a94a-462a-845c-86027f625306
Container State: QUASI_CLOSED
Datanodes: [006fd0ff-a52c-4bdd-a888-e3a005828696/localhost,
2260916d-5e5d-40a5-b1b4-a4e706862f8a/localhost]
```
- Test by help of debugger
Break while writing file. Stop the leader datanode of the writing pipeline, client will request a new block to scm through om successfully.
Modify the hadoop-env.sh
```bash
export HADOOP_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5004"
```
put a bigger file more than 256MB
```
bin/ozone sh key put -r TWO /myvol/mybucket/spark-tpc-ds-performance-test-master2.zip ~/Downloads/spark-tpc-ds-performance-test-master.zip
```
set a break point to the line of `updateFlushLength` in BlockOutputStream#write method, when break hit two times, it mean there are some chunk persist to the datanode. find the current pipeline's leader and stop it.
```bash
bin/ozone --daemon stop datanode
```
resume the debugger to let the process run, check the file write to the Ozone successfully.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong closed pull request #983: HDDS-3584. Support write 2 replication
Posted by GitBox <gi...@apache.org>.
maobaolong closed pull request #983:
URL: https://github.com/apache/hadoop-ozone/pull/983
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #983: HDDS-3584. Support write 2 replication
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #983:
URL: https://github.com/apache/hadoop-ozone/pull/983#issuecomment-681824767
Lets close this PR for the discussion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #983: HDDS-3584. Support write 2 replication
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #983:
URL: https://github.com/apache/hadoop-ozone/pull/983#issuecomment-651606953
What is the plan with this PR. @arp7 do you have any concerns? Do we need more discussion?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #983: HDDS-3584. Support write 2 replication
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #983:
URL: https://github.com/apache/hadoop-ozone/pull/983#issuecomment-659297817
/pending more discussion is requested by @arp7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org