[jira] [Created] (KUDU-1970) Integration test for data scalability

["Adar Dembo (JIRA)" <ji...@apache.org>]

Adar Dembo created KUDU-1970:
--------------------------------

             Summary: Integration test for data scalability
                 Key: KUDU-1970
                 URL: https://issues.apache.org/jira/browse/KUDU-1970
             Project: Kudu
          Issue Type: Sub-task
          Components: master, tserver
    Affects Versions: 1.4.0
            Reporter: Adar Dembo
            Assignee: Adar Dembo


To help test data scalability fixes, we need a way to easily produce an environment that exhibits our current scalability issues. I'm sure one of our long-running workloads would be up to the task, but aside from taking a long time, it'd also fill up the disk, which makes it unusable on most developer machines. Ultimately, data isn't really the root cause of our scalability woes; it's the metadata necessary to maintain the data that hurts us. So an idealized environment would be heavy on the metadata. Here's a not-so-exhaustive list:
* Many tablets.
* Many columns per tablet.
* Many rowsets per tablet.
* Many data blocks.
* Many tables (tservers don't care about this, but maybe the master does?)

Let's write an integration test that swamps the machine with the above. It should be use an external mini cluster to simplify isolating master and tserver performance characteristics, but it needn't have more than one instance of each.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)