[GitHub] [fluo-muchos] keith-turner edited a comment on issue #267: Change Storage Size for Data

[GitBox <gi...@apache.org>]

keith-turner edited a comment on issue #267: Change Storage Size for Data
URL: https://github.com/apache/fluo-muchos/issues/267#issuecomment-514657171
 
 
   > As apart of my use case, it is requiring that I ingest close to 5TB of total data. So, as apart of that, I am trying to increase the storage capacity for Accumulo and Hadoop nodes.
   
   Muchos will use all of the emphemeral storage available for the instance types it knows about (more on this later).   To get more storage, I would suggest the changing the following muchos settings in `muchos.props` to use [instance types](https://aws.amazon.com/ec2/instance-types/) with more storage.  The settings below would make your leader nodes use `m5d.xlarge` instance types which have a small amount of ephemeral.  Your worker nodes would use `d2.xlarge` instances types which have 3x2TB of ephemeral.
   
   ```
   default_instance_type = m5d.xlarge
   worker_instance_type = d2.xlarge
   ```
   
   The `d2` instance types have slower disk, if you want fast storage consider the `i3` instance types.
   
   For muchos to use all of a instances types local drives it has to know about them.  Currently this information is hard coded in muchos at [util.py line 45](https://github.com/apache/fluo-muchos/blob/80281f2a9864d59c97e762dfcdc6ba7f44c47e5e/lib/muchos/util.py#L45).  The python map is used in the method `get_ephemeral_devices` which you has a special case for `m5d` instance types at [util.py line 115](https://github.com/apache/fluo-muchos/blob/80281f2a9864d59c97e762dfcdc6ba7f44c47e5e/lib/muchos/util.py#L115).
   
   > Lastly, I am hoping to identify if there is an option to use EBS Volumes vs Ephemeral Storage for (long-term storage and backups)
   
   There has been no effort to automate setting things up on EBS.  One of the reasons for this is that Muchos was created to automate running short term test and long term persistence was not a concern.  I would welcome anyone who is interested in making these types of changes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services