You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by "Abdullah Alamoudi (JIRA)" <ji...@apache.org> on 2016/03/10 15:39:40 UTC
[jira] [Assigned] (ASTERIXDB-1337) Dataset Memory Management on
Multi-Partition NC
[ https://issues.apache.org/jira/browse/ASTERIXDB-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abdullah Alamoudi reassigned ASTERIXDB-1337:
--------------------------------------------
Assignee: Michael Blow
> Dataset Memory Management on Multi-Partition NC
> -----------------------------------------------
>
> Key: ASTERIXDB-1337
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1337
> Project: Apache AsterixDB
> Issue Type: Improvement
> Components: AsterixDB, Storage
> Reporter: Murtadha Hubail
> Assignee: Michael Blow
>
> Currently, each dataset has a fixed memory budget - total virtual buffer cache (VBC) budget - which is configurable by the following attributes in asterix configuration file:
> storage.memorycomponent.pagesize (Default 128K)
> storage.memorycomponent.numpages (Default 256 pages)
> Note: a different attributes are used for Metadata datasets.
> During query compilation, any index that will be accessed uses AbstractLSMIndexDataflowHelperFactory which is passed an instance of AsterixVirtualBufferCacheProvider.
> Each dataset has a single (AsterixVirtualBufferCacheProvider), which makes all indexes and their partitions (different IO devices) of this dataset on the same node get access to the same dataset VBC.
> During runtime, when the AbstractLSMIndexDataflowHelperFactory is used to create the actual IndexDataflowHelper, the dataset VBC is initialized. The total VBC budget of the dataset is divided into a number of VBCs which is configurable in asterix configuration file as:
> storage.memorycomponent.numcomponents (Default 2 VBCs)
> Each one of those VBCs is created as an object of type MultitenantVirtualBufferCache (MVBC) (in DatasetLifecycleManager#initializeDatasetVirtualBufferCache). The size of each of these MVBC is (storage.memorycomponent.numpages / storage.memorycomponent.numcomponents). Even though the dataset VBCs have been initialized, no memory is allocated yet. This avoids the memory allocation of disk read only queries or bulkload DDLs.
> Upon the first modification (on LSMHarness#modify) of any index partition that belongs to this dataset, we allocate the memory on all MVBCs that we initialized earlier. This makes all indexes and their partitions of the dataset on the same node compete on the budget of a single MVBC at a time. Once a MVBC is full, all files opened in it are scheduled to be flushed, and we switch to another MVBC (if any is available). This will have the effect of making frequent flushes of many small files, which will lead to frequent merges.
> I think that it would be better if each partition (IO device on the node) has its own MVBC budget.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)