You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/10/25 12:21:46 UTC

[GitHub] [accumulo] dlmarion opened a new issue, #3040: Does Accumulo support Erasure Coding for its files?

dlmarion opened a new issue, #3040:
URL: https://github.com/apache/accumulo/issues/3040

   Installed Hadoop 3.3.4 with minimal configuration changes. The NameNode shows that the enabled erasure coding policy is RS-6-3-1024k. After init'ing Accumulo, the RFiles show as replicated.
   ```
   hadoop fs -ls -e /accumulo/tables/\!0/table_info
   Found 1 items
   -rw-r--r--   3 hadoop supergroup Replicated        265 2022-10-25 11:56 /accumulo/tables/!0/table_info/0_1.rf
   ```
   I'm curious if the way the files are created enforce a block replication approach vs a block striping approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] ctubbsii commented on issue #3040: Does Accumulo support Erasure Coding for its files?

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on issue #3040:
URL: https://github.com/apache/accumulo/issues/3040#issuecomment-1290463008

   I believe Accumulo has a per-table configuration to specify the number of replicas. I do not know much about the API/configuration options for HDFS's erasure coding, but I imagine that you can set the number of replicas to 1 in the Accumulo configuration, and enable erasure coding for HDFS through it's own configuration. Getting this working as an example, may be a useful exercise for a blog post for our website.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] dlmarion commented on issue #3040: Does Accumulo support Erasure Coding for its files?

Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3040:
URL: https://github.com/apache/accumulo/issues/3040#issuecomment-1290479157

   It appears that the `RS-6-3-1024k` policy is available for use, but not set on any directory.
   
   ```
   hdfs ec -getPolicy -path /accumulo
   The erasure coding policy of /accumulo is unspecified
   hdfs ec -getPolicy -path /
   The erasure coding policy of / is unspecified
   ```
   
   We'll need to investigate further.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] dlmarion commented on issue #3040: Does Accumulo support Erasure Coding for its files?

Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3040:
URL: https://github.com/apache/accumulo/issues/3040#issuecomment-1290625838

   Reference: https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] dlmarion commented on issue #3040: Does Accumulo support Erasure Coding for its files?

Posted by GitBox <gi...@apache.org>.
dlmarion commented on issue #3040:
URL: https://github.com/apache/accumulo/issues/3040#issuecomment-1299013736

   I think the answer is [here](https://accumulo.apache.org/docs/2.x/administration/erasure-coding). I didn't see this until after the 2.1.0 release documentation was merged and put up onto the staging site.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [accumulo] dlmarion closed issue #3040: Does Accumulo support Erasure Coding for its files?

Posted by GitBox <gi...@apache.org>.
dlmarion closed issue #3040: Does Accumulo support Erasure Coding for its files?
URL: https://github.com/apache/accumulo/issues/3040


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org