You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@asterixdb.apache.org by Jianfeng Jia <ji...@gmail.com> on 2016/03/10 23:46:41 UTC
Should in-memory components from different dataset share the entire memory?
Dear Devs,
I have some questions about the memory management of the in-memory components for different datasets.
The current AsterixDB backing the cloudberry demo is down every few days. It always throws an exception like following:
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Failed to open index with resource ID 7 since it does not exist.
As described in ASTERIXDB-1337, each dataset has a fixed budget no matter how small/big it is. Then the number of datasets can be loaded at the same time is also fixed by $number = storage.memorycomponent.globalbudget/storage.memorycomponent.numpages. My question is if we have more than $number of datasets, then the eviction will happen? Will it evict a entire dataset of the victim? Base on the symptom of above exception, it seems the metadata get evicted? Could we protect the metadata from eviction?
A more fundament question Is it possible that all those datasets share a global budget in a multi-tenant way?
In my workload there are one main dataset( ~10Gb) and five tiny auxiliary datasets (each size <20M). In addition, the client will create a bunch of temporary datasets depends on how many concurrent users are and each temp-dataset will be “refreshed" for a new query. (The refresh is done by drop and create the temp-dataset). It’s hard to find one storage.memorycomponent.numpages that make every dataset happy.
Best,
Jianfeng Jia
PhD Candidate of Computer Science
University of California, Irvine
Re: Should in-memory components from different dataset share the entire memory?
Posted by Jianfeng Jia <ji...@gmail.com>.
Got it. Thanks Murtadha.
> On Mar 10, 2016, at 2:55 PM, Murtadha Hubail <hu...@gmail.com> wrote:
>
> Hi Jianfeng,
>
> I just want to clarify that the issue causing the exception is the fact metadata datasets are being evicted and not opened again.
> This is caused by the special treat of metadata indexes (not going thru the same code path as other indexes). I have already filed an issue[1] for it and will submit a fix today that will fix the exception issue.
>
> The dynamic memory allocation needs its separate discussion.
>
> Cheers,
> Murtadha
>
> [1] https://issues.apache.org/jira/browse/ASTERIXDB-1338 <https://issues.apache.org/jira/browse/ASTERIXDB-1338>
>
>> On Mar 10, 2016, at 2:46 PM, Jianfeng Jia <ji...@gmail.com> wrote:
>>
>> Dear Devs,
>>
>> I have some questions about the memory management of the in-memory components for different datasets.
>>
>> The current AsterixDB backing the cloudberry demo is down every few days. It always throws an exception like following:
>> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Failed to open index with resource ID 7 since it does not exist.
>>
>> As described in ASTERIXDB-1337, each dataset has a fixed budget no matter how small/big it is. Then the number of datasets can be loaded at the same time is also fixed by $number = storage.memorycomponent.globalbudget/storage.memorycomponent.numpages. My question is if we have more than $number of datasets, then the eviction will happen? Will it evict a entire dataset of the victim? Base on the symptom of above exception, it seems the metadata get evicted? Could we protect the metadata from eviction?
>>
>> A more fundament question Is it possible that all those datasets share a global budget in a multi-tenant way?
>> In my workload there are one main dataset( ~10Gb) and five tiny auxiliary datasets (each size <20M). In addition, the client will create a bunch of temporary datasets depends on how many concurrent users are and each temp-dataset will be “refreshed" for a new query. (The refresh is done by drop and create the temp-dataset). It’s hard to find one storage.memorycomponent.numpages that make every dataset happy.
>>
>>
>>
>> Best,
>>
>> Jianfeng Jia
>> PhD Candidate of Computer Science
>> University of California, Irvine
>>
>
Best,
Jianfeng Jia
PhD Candidate of Computer Science
University of California, Irvine
Re: Should in-memory components from different dataset share the entire memory?
Posted by Murtadha Hubail <hu...@gmail.com>.
Hi Jianfeng,
I just want to clarify that the issue causing the exception is the fact metadata datasets are being evicted and not opened again.
This is caused by the special treat of metadata indexes (not going thru the same code path as other indexes). I have already filed an issue[1] for it and will submit a fix today that will fix the exception issue.
The dynamic memory allocation needs its separate discussion.
Cheers,
Murtadha
[1] https://issues.apache.org/jira/browse/ASTERIXDB-1338 <https://issues.apache.org/jira/browse/ASTERIXDB-1338>
> On Mar 10, 2016, at 2:46 PM, Jianfeng Jia <ji...@gmail.com> wrote:
>
> Dear Devs,
>
> I have some questions about the memory management of the in-memory components for different datasets.
>
> The current AsterixDB backing the cloudberry demo is down every few days. It always throws an exception like following:
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Failed to open index with resource ID 7 since it does not exist.
>
> As described in ASTERIXDB-1337, each dataset has a fixed budget no matter how small/big it is. Then the number of datasets can be loaded at the same time is also fixed by $number = storage.memorycomponent.globalbudget/storage.memorycomponent.numpages. My question is if we have more than $number of datasets, then the eviction will happen? Will it evict a entire dataset of the victim? Base on the symptom of above exception, it seems the metadata get evicted? Could we protect the metadata from eviction?
>
> A more fundament question Is it possible that all those datasets share a global budget in a multi-tenant way?
> In my workload there are one main dataset( ~10Gb) and five tiny auxiliary datasets (each size <20M). In addition, the client will create a bunch of temporary datasets depends on how many concurrent users are and each temp-dataset will be “refreshed" for a new query. (The refresh is done by drop and create the temp-dataset). It’s hard to find one storage.memorycomponent.numpages that make every dataset happy.
>
>
>
> Best,
>
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
>
Re: Should in-memory components from different dataset share the
entire memory?
Posted by Mike Carey <dt...@gmail.com>.
There is still the broader question - as the overall memory of a node
(NC) has to be shared among the three kinds of memory (at least) that I
also mentioned just a minute ago:
- buffer cache (global/shared memory for caching pages of disk
components)
- in-memory components (i.e., the current thread's topic)
- working memory (needed for hash-based and sort-based operators like
joins/aggs)
Whatever we do next should probably start keeping in mind this
bigger-picture/zero-sum game.
On 3/10/16 3:23 PM, Jianfeng Jia wrote:
> Another way is to allocate the entire global space upfront, but there is no upper-bound for each dataset. The bigger dataset gets more pages. The drawback is that we may waste more space if all dataset are all small.
>
> The ideal case is there is only one dynamically memory allocation manager with one global upper-bound, and all datasets can share the space without an extra bound per dataset.
>
>
>> On Mar 10, 2016, at 2:55 PM, Yingyi Bu <bu...@gmail.com> wrote:
>>
>>>> A more fundament question Is it possible that all those datasets share a
>> global budget in a multi-tenant way?
>> In principle, the budget should just be a upper-bound. If a dataset doesn't
>> need that much, it shouldn't pre-allocate all
>> "storage.memorycomponent.numpages"
>> pages.
>>
>> However, in the current implementation, we pre-allocate all in-memory pages
>> upfront:
>> https://github.com/apache/incubator-asterixdb-hyracks/blob/master/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/VirtualBufferCache.java#L247
>>
>> I think we should fix it to dynamically allocate memory when needed. (Disk
>> buffer cache already does that.)
>>
>> Best,
>> Yingyi
>>
>>
>> On Thu, Mar 10, 2016 at 2:46 PM, Jianfeng Jia <ji...@gmail.com>
>> wrote:
>>
>>> Dear Devs,
>>>
>>> I have some questions about the memory management of the in-memory
>>> components for different datasets.
>>>
>>> The current AsterixDB backing the cloudberry demo is down every few days.
>>> It always throws an exception like following:
>>> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Failed
>>> to open index with resource ID 7 since it does not exist.
>>>
>>> As described in ASTERIXDB-1337, each dataset has a fixed budget no matter
>>> how small/big it is. Then the number of datasets can be loaded at the same
>>> time is also fixed by $number =
>>> storage.memorycomponent.globalbudget/storage.memorycomponent.numpages. My
>>> question is if we have more than $number of datasets, then the eviction
>>> will happen? Will it evict a entire dataset of the victim? Base on the
>>> symptom of above exception, it seems the metadata get evicted? Could we
>>> protect the metadata from eviction?
>>>
>>> A more fundament question Is it possible that all those datasets share a
>>> global budget in a multi-tenant way?
>>> In my workload there are one main dataset( ~10Gb) and five tiny auxiliary
>>> datasets (each size <20M). In addition, the client will create a bunch of
>>> temporary datasets depends on how many concurrent users are and each
>>> temp-dataset will be “refreshed" for a new query. (The refresh is done by
>>> drop and create the temp-dataset). It’s hard to find one
>>> storage.memorycomponent.numpages that make every dataset happy.
>>>
>>>
>>>
>>> Best,
>>>
>>> Jianfeng Jia
>>> PhD Candidate of Computer Science
>>> University of California, Irvine
>>>
>>>
>
>
> Best,
>
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
>
>
Re: Should in-memory components from different dataset share the entire memory?
Posted by Jianfeng Jia <ji...@gmail.com>.
Another way is to allocate the entire global space upfront, but there is no upper-bound for each dataset. The bigger dataset gets more pages. The drawback is that we may waste more space if all dataset are all small.
The ideal case is there is only one dynamically memory allocation manager with one global upper-bound, and all datasets can share the space without an extra bound per dataset.
> On Mar 10, 2016, at 2:55 PM, Yingyi Bu <bu...@gmail.com> wrote:
>
>>> A more fundament question Is it possible that all those datasets share a
> global budget in a multi-tenant way?
> In principle, the budget should just be a upper-bound. If a dataset doesn't
> need that much, it shouldn't pre-allocate all
> "storage.memorycomponent.numpages"
> pages.
>
> However, in the current implementation, we pre-allocate all in-memory pages
> upfront:
> https://github.com/apache/incubator-asterixdb-hyracks/blob/master/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/VirtualBufferCache.java#L247
>
> I think we should fix it to dynamically allocate memory when needed. (Disk
> buffer cache already does that.)
>
> Best,
> Yingyi
>
>
> On Thu, Mar 10, 2016 at 2:46 PM, Jianfeng Jia <ji...@gmail.com>
> wrote:
>
>> Dear Devs,
>>
>> I have some questions about the memory management of the in-memory
>> components for different datasets.
>>
>> The current AsterixDB backing the cloudberry demo is down every few days.
>> It always throws an exception like following:
>> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Failed
>> to open index with resource ID 7 since it does not exist.
>>
>> As described in ASTERIXDB-1337, each dataset has a fixed budget no matter
>> how small/big it is. Then the number of datasets can be loaded at the same
>> time is also fixed by $number =
>> storage.memorycomponent.globalbudget/storage.memorycomponent.numpages. My
>> question is if we have more than $number of datasets, then the eviction
>> will happen? Will it evict a entire dataset of the victim? Base on the
>> symptom of above exception, it seems the metadata get evicted? Could we
>> protect the metadata from eviction?
>>
>> A more fundament question Is it possible that all those datasets share a
>> global budget in a multi-tenant way?
>> In my workload there are one main dataset( ~10Gb) and five tiny auxiliary
>> datasets (each size <20M). In addition, the client will create a bunch of
>> temporary datasets depends on how many concurrent users are and each
>> temp-dataset will be “refreshed" for a new query. (The refresh is done by
>> drop and create the temp-dataset). It’s hard to find one
>> storage.memorycomponent.numpages that make every dataset happy.
>>
>>
>>
>> Best,
>>
>> Jianfeng Jia
>> PhD Candidate of Computer Science
>> University of California, Irvine
>>
>>
Best,
Jianfeng Jia
PhD Candidate of Computer Science
University of California, Irvine
Re: Should in-memory components from different dataset share the
entire memory?
Posted by Mike Carey <dt...@gmail.com>.
+1 for this incremental allocation proposal. Also, our whole budgeting
/ memory management model was version 0 - and we need to think about
version 1 sometime in the not-so-distant future. (Once upon a time we
imagined an intelligent memory controller overseeing the union of buffer
cache, working memories for queries, and in-memory components, and
working with a future version of the query optimizer to make more
intelligent choices. In the absence of statistics, workload info, and
cost info, or runtime observations thereof, this was our first
brain-dead approach to getting something running that we could improve
later - and it seems to be getting towards later now. :-))
On 3/10/16 2:55 PM, Yingyi Bu wrote:
>>> A more fundament question Is it possible that all those datasets share a
> global budget in a multi-tenant way?
> In principle, the budget should just be a upper-bound. If a dataset doesn't
> need that much, it shouldn't pre-allocate all
> "storage.memorycomponent.numpages"
> pages.
>
> However, in the current implementation, we pre-allocate all in-memory pages
> upfront:
> https://github.com/apache/incubator-asterixdb-hyracks/blob/master/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/VirtualBufferCache.java#L247
>
> I think we should fix it to dynamically allocate memory when needed. (Disk
> buffer cache already does that.)
>
> Best,
> Yingyi
>
>
> On Thu, Mar 10, 2016 at 2:46 PM, Jianfeng Jia <ji...@gmail.com>
> wrote:
>
>> Dear Devs,
>>
>> I have some questions about the memory management of the in-memory
>> components for different datasets.
>>
>> The current AsterixDB backing the cloudberry demo is down every few days.
>> It always throws an exception like following:
>> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Failed
>> to open index with resource ID 7 since it does not exist.
>>
>> As described in ASTERIXDB-1337, each dataset has a fixed budget no matter
>> how small/big it is. Then the number of datasets can be loaded at the same
>> time is also fixed by $number =
>> storage.memorycomponent.globalbudget/storage.memorycomponent.numpages. My
>> question is if we have more than $number of datasets, then the eviction
>> will happen? Will it evict a entire dataset of the victim? Base on the
>> symptom of above exception, it seems the metadata get evicted? Could we
>> protect the metadata from eviction?
>>
>> A more fundament question Is it possible that all those datasets share a
>> global budget in a multi-tenant way?
>> In my workload there are one main dataset( ~10Gb) and five tiny auxiliary
>> datasets (each size <20M). In addition, the client will create a bunch of
>> temporary datasets depends on how many concurrent users are and each
>> temp-dataset will be “refreshed" for a new query. (The refresh is done by
>> drop and create the temp-dataset). It’s hard to find one
>> storage.memorycomponent.numpages that make every dataset happy.
>>
>>
>>
>> Best,
>>
>> Jianfeng Jia
>> PhD Candidate of Computer Science
>> University of California, Irvine
>>
>>
Re: Should in-memory components from different dataset share the
entire memory?
Posted by Yingyi Bu <bu...@gmail.com>.
>> A more fundament question Is it possible that all those datasets share a
global budget in a multi-tenant way?
In principle, the budget should just be a upper-bound. If a dataset doesn't
need that much, it shouldn't pre-allocate all
"storage.memorycomponent.numpages"
pages.
However, in the current implementation, we pre-allocate all in-memory pages
upfront:
https://github.com/apache/incubator-asterixdb-hyracks/blob/master/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/VirtualBufferCache.java#L247
I think we should fix it to dynamically allocate memory when needed. (Disk
buffer cache already does that.)
Best,
Yingyi
On Thu, Mar 10, 2016 at 2:46 PM, Jianfeng Jia <ji...@gmail.com>
wrote:
> Dear Devs,
>
> I have some questions about the memory management of the in-memory
> components for different datasets.
>
> The current AsterixDB backing the cloudberry demo is down every few days.
> It always throws an exception like following:
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Failed
> to open index with resource ID 7 since it does not exist.
>
> As described in ASTERIXDB-1337, each dataset has a fixed budget no matter
> how small/big it is. Then the number of datasets can be loaded at the same
> time is also fixed by $number =
> storage.memorycomponent.globalbudget/storage.memorycomponent.numpages. My
> question is if we have more than $number of datasets, then the eviction
> will happen? Will it evict a entire dataset of the victim? Base on the
> symptom of above exception, it seems the metadata get evicted? Could we
> protect the metadata from eviction?
>
> A more fundament question Is it possible that all those datasets share a
> global budget in a multi-tenant way?
> In my workload there are one main dataset( ~10Gb) and five tiny auxiliary
> datasets (each size <20M). In addition, the client will create a bunch of
> temporary datasets depends on how many concurrent users are and each
> temp-dataset will be “refreshed" for a new query. (The refresh is done by
> drop and create the temp-dataset). It’s hard to find one
> storage.memorycomponent.numpages that make every dataset happy.
>
>
>
> Best,
>
> Jianfeng Jia
> PhD Candidate of Computer Science
> University of California, Irvine
>
>