You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "z-bb (via GitHub)" <gi...@apache.org> on 2023/06/19 08:16:28 UTC
[GitHub] [ozone] z-bb opened a new pull request, #4929: HDDS-8883. Add carry ecPolicyName method.
z-bb opened a new pull request, #4929:
URL: https://github.com/apache/ozone/pull/4929
## What changes were proposed in this pull request?
Add carry ecPolicyName method.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/projects/HDDS/issues/HDDS-8883?filter=myopenissues
## How was this patch tested?
ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] z-bb commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "z-bb (via GitHub)" <gi...@apache.org>.
z-bb commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1236731702
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
@sodonnel Thank you for your reply.
In response to this question The data is EC in Ozone, but goes to a non EC folder in HDFS - how will that work, and what will the replication factor be set to in HDFS? The result of my test is 3.
At this jira https://issues.apache.org/jira/browse/HADOOP-14254 add a "-pe" switch to ensure erasure coded files at source are copied as erasure coded files at destination.
But when I distcp specify -p HDFS EC to Ozone Ratis Bucket, The number of replicas in ozone is neither ec nor RATIS-THREE, but Ratis-ONE.
In this pr https://github.com/apache/hadoop/commit/c757cb61ebc9e69d9f6f143da91189b9f0517ee9#diff-6207b5e70af95b856263703af631c577358dd06662d364f06347827d92f734f2. His conditions are very harsh, `sourceStatus instanceof HdfsFileStatus && targetFS instanceof DistributedFileSystem`
Only then will the ecPolicy attribute be obtained, which causes distcp to cannot obtain the ecPolicy in the ozone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1235109893
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
> Forcibly convert fs to RootedOzoneFileSystem in hadoop ... similar to forcibly converting fs to DistributedFileSystem
I don't think this will work. Hadoop has no dependency on Ozone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] z-bb commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "z-bb (via GitHub)" <gi...@apache.org>.
z-bb commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1236206236
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
@adoroszlai At present, all I can think of is to add an interface in the ozone, and then pass the ecPolicyName attribute to the ozone on the hadoop side. Do you think there are other solutions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] z-bb commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "z-bb (via GitHub)" <gi...@apache.org>.
z-bb commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1233880552
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
@sodonnel Thanks for review, The ecPolicyName attribute of ec is added in this pr https://github.com/apache/hadoop/commit/c757cb61ebc9e69d9f6f143da91189b9f0517ee9，ReplicationConfig is a class of ozone, not hadoop. In the next step, I will add an interface call to hadoop and pass ecPolicyName to ozone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-8883. Add carry ecPolicyName method. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai closed pull request #4929: HDDS-8883. Add carry ecPolicyName method.
URL: https://github.com/apache/ozone/pull/4929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] z-bb commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "z-bb (via GitHub)" <gi...@apache.org>.
z-bb commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1236758432
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
@sodonnel As far as I understand, ozone is not extend DistributedFileSystem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1236751353
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
I see that https://issues.apache.org/jira/browse/HDFS-10996 added a "per file" EC setting to the `hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java` class - does Ozone already extend this class, and if so, perhaps the HDFS interface is already is place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] z-bb commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "z-bb (via GitHub)" <gi...@apache.org>.
z-bb commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1233880552
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
@sodonnel Thanks for review, The ecPolicyName attribute of ec is added in this pr https://github.com/apache/hadoop/commit/c757cb61ebc9e69d9f6f143da91189b9f0517ee9, ReplicationConfig is a class of ozone, not hadoop. In the next step, I will add an interface call to hadoop and pass ecPolicyName to ozone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1233866018
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
I think the new interface should be:
```
OzoneFSOutputStream createFile(String key, ReplicationConfig repConfig,
boolean overWrite, boolean recursive) throws IOException;
```
This includes replication and EC definitions in a single variable, and is more consistent with what we have done in other places throughout Ozone.
At the point in distcp where it calls into Ozone, is it via this interface? Is it possible to create the replicationConfig object there depending on the files replication or EC definition in distcp?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] z-bb commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "z-bb (via GitHub)" <gi...@apache.org>.
z-bb commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1233895410
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
@sodonnel My idea is to keep all the previous parameters and only add an ecPolicyName attribute. Maybe the previous parameters will be useful later. Forcibly convert fs to RootedOzoneFileSystem in hadoop, and then call the createFile method, which is similar to forcibly converting fs to DistributedFileSystem above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] z-bb commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "z-bb (via GitHub)" <gi...@apache.org>.
z-bb commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1237805494
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
> Ozone extends FileSystem, and it looks like distcp either talks to a DistributedFileSystem (HDFS) or a FileSystem. Therefore I suspect we need to change the FileSystem class in Hadoop to have the new parameter first. Otherwise, even though we have added the new parameters here, distcp will not be able to see them, as it only knows about Filesystem. We cannot cast an object in distcp into an Ozone class.
@sodonnel I think this is possible, cc @ChenSammi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1237712056
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
Ozone extends FileSystem, and it looks like distcp either talks to a DistributedFileSystem (HDFS) or a FileSystem. Therefore I suspect we need to change the FileSystem class in Hadoop to have the new parameter first. Otherwise, even though we have added the new parameters here, distcp will not be able to see them, as it only knows about Filesystem. We cannot cast an object in distcp into an Ozone class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] z-bb commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "z-bb (via GitHub)" <gi...@apache.org>.
z-bb commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1236731702
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
@sodonnel Thank you for your reply.
In response to this question The data is EC in Ozone, but goes to a non EC folder in HDFS - how will that work, and what will the replication factor be set to in HDFS? The result of my test is 3.
At this jira https://issues.apache.org/jira/browse/HADOOP-14254 add a "-pe" switch to ensure erasure coded files at source are copied as erasure coded files at destination.
But when I distcp specify -p HDFS EC to Ozone Ratis Bucket, The number of replicas in ozone is neither ec nor RATIS-THREE, but Ratis-ONE.
In this pr https://github.com/apache/hadoop/commit/c757cb61ebc9e69d9f6f143da91189b9f0517ee9#diff-6207b5e70af95b856263703af631c577358dd06662d364f06347827d92f734f2. His conditions are very harsh, `sourceStatus instanceof HdfsFileStatus` and `targetFS instanceof DistributedFileSystem`
Only then will the ecPolicy attribute be obtained, which causes distcp to cannot obtain the ecPolicy in the ozone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] z-bb commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "z-bb (via GitHub)" <gi...@apache.org>.
z-bb commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1237805494
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
> Ozone extends FileSystem, and it looks like distcp either talks to a DistributedFileSystem (HDFS) or a FileSystem. Therefore I suspect we need to change the FileSystem class in Hadoop to have the new parameter first. Otherwise, even though we have added the new parameters here, distcp will not be able to see them, as it only knows about Filesystem. We cannot cast an object in distcp into an Ozone class.
@sodonnel I think this is possible, cc @adoroszlai @ChenSammi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on a diff in pull request #4929: HDDS-8883. Add carry ecPolicyName method.
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #4929:
URL: https://github.com/apache/ozone/pull/4929#discussion_r1236614774
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java:
##########
@@ -46,9 +46,17 @@ public interface OzoneClientAdapter {
OzoneFSOutputStream createFile(String key, short replication,
boolean overWrite, boolean recursive) throws IOException;
+ OzoneFSOutputStream createFile(String key, short replication,
+ boolean overWrite, boolean recursive,
+ String ecPolicyName) throws IOException;
Review Comment:
I'd be interested to see how this looks at the HDFS side. The problem is that the HDFS defined interface does not allow for passing an EC replication definition. It only has "int replication" in the interface.
For HDFS, if you try to write to a folder which has an EC policy set, HDFS detects the policy on the folder and ignores the replication field, writing EC. There is no way to set EC for a specific key in HDFS, and you cannot write replicated data into an EC enabled folder either.
Ozone is different. We allow the replication config to be inherited from the bucket if nothing is passed for the key. If something is passed for the key, it overrides whatever is set on the bucket. So you have have an EC-3-2 bucket, and write RATIS-THREE data into it.
There is the possibility of this being a two way problem - what if someone uses distcp to copy data from Ozone to HDFS?
* The data is EC in Ozone, but goes to a non EC folder in HDFS - how will that work, and what will the replication factor be set to in HDFS?
* The data is Ratis in Ozone, going to an EC folder in HDFS - this is easy, HDFS will set the policy to EC.
Then we have the HDFS to Ozone scenarios:
* HDFS EC to Ozone EC bucket
* HDFS EC to Ozone Ratis Bucket
* HDFS Replicated to Ozone EC bucket
* HDFS Replicated to Ozone Replicated
I am not sure where in distcp it calls to Ozone, but perhaps we need to change the HDFS interface that Ozone implements first to define how this can work from distcp, and then fix Ozone to make it use that extended interface?
I wonder if there is some flag we could pass, or a special replication value (eg -1) to make Ozone ignore the key level setting and just use whatever is set at the bucket? Although even then, we would not want HDFS EC data to end up as Ratis-ONE data in Ozone if a bucket is RATIS-THREE.
Could you point us at the code in distcp that would need to change to use a new interface, and then we can have a think about any possible ways of changing it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org