Is it OK for Hyracks to access the config option?

[Taewoo Kim <wa...@gmail.com>]

Hello All,

Here is the background. I'm currently fixing an inverted-index search OOM
issue. This could happen because the core inverted-index search module does
not have any budget limit. One might ask that B+-tree and R-tree search
doesn't have any budget limit either. However, a difference is that an
inverted-index has an inverted list (list of primary keys per secondary
key). That is, for a single PK, there can be multiple secondary keys. So,
dealing with this requires additional frames. So, let's assume that we need
to have multiple frames to answer a text query that utilizes an inverted
index.

Here is my issue. We will add one more parameter to the config (e.g.,
compiler.textsearchmemory = 32MB) to control this behavior. The
index-search core module needs to receive and conform to this budget
information. For inverted-index search query case, it's relatively easy. I
can use HyracksTaskContext to pass that since there's a pipeline from
compiler to the search module (IndexSearchOperatorNodePushable). However, I
found an issue regarding the merge case. Merge of an LSM index happens in
Hyracks codebase and index merge code doesn't know about
HyracksTaskContext. So, I talked to Abdullah and he shared a hint about
ServiceContext. In the ServiceContext, there is AppConfig interface and
this contains all config information. And when creating an LSM index
instance, this ServiceContext is accessible.

So, my final question is: is it OK to access AppConfig from Hyracks code by
specifying the exact parameter (e.g., compiler.textsearchmemory) to get
some information? Technically, it's doable since the code exists in
Hyracks. However, most similar parameters are processed in Asterix
codebase, not in  Hyracks codebase. If not, what could be a way to ensure
this?

Thanks for reading!

Best,
Taewoo

Re: Is it OK for Hyracks to access the config option?

[Ian Maxon <im...@uci.edu>]

The ini-style config should be fine to access from Hyracks-level, it's
modular. Some of the properties are already accessed from within
Hyracks for the most part (e.g. IO devices, etc).

On Tue, Oct 24, 2017 at 1:20 PM, Taewoo Kim <wa...@gmail.com> wrote:
> Never mind about the first question. The problem becomes simpler (but still
> couldn't find the solution) after talking to Chen Luo. Thanks.
>
> Let's say the inverted index search module Blah1 (Actually SearchResult and
> FixedSizeElementInvertedListCursor in case you are interested in). For this
> Blah1 module to generate intermediate and final result work properly, the
> caller needs to pass three parameters (frameSize, how to
> allocate/deallocate buffers, and the maximum number of frames that Blah1
> can have). Blah1 can be called in two ways.
>
> 1) Search using HyracksTaskCotext (class name: Task)
> 2) Search initiated by merge (class name: AsynchronousScheduler)
>
> 1) and 2) are independent processes so it looks like hard to associate 1)
> in 2) or 2) in 1). So, for Blah1, it needs to consider 1) as default and 2)
> as a special case or vice versa. Any suggestions? Again, for B+ tree, R
> tree, this problem doesn't exist since there is no intermediate result
> calculation.
>
>
>
> Best,
> Taewoo
>
> On Tue, Oct 24, 2017 at 9:21 AM, Taewoo Kim <wa...@gmail.com> wrote:
>
>> Hello All,
>>
>> Here is the background. I'm currently fixing an inverted-index search OOM
>> issue. This could happen because the core inverted-index search module does
>> not have any budget limit. One might ask that B+-tree and R-tree search
>> doesn't have any budget limit either. However, a difference is that an
>> inverted-index has an inverted list (list of primary keys per secondary
>> key). That is, for a single PK, there can be multiple secondary keys. So,
>> dealing with this requires additional frames. So, let's assume that we need
>> to have multiple frames to answer a text query that utilizes an inverted
>> index.
>>
>> Here is my issue. We will add one more parameter to the config (e.g.,
>> compiler.textsearchmemory = 32MB) to control this behavior. The
>> index-search core module needs to receive and conform to this budget
>> information. For inverted-index search query case, it's relatively easy. I
>> can use HyracksTaskContext to pass that since there's a pipeline from
>> compiler to the search module (IndexSearchOperatorNodePushable). However,
>> I found an issue regarding the merge case. Merge of an LSM index happens in
>> Hyracks codebase and index merge code doesn't know about
>> HyracksTaskContext. So, I talked to Abdullah and he shared a hint about
>> ServiceContext. In the ServiceContext, there is AppConfig interface and
>> this contains all config information. And when creating an LSM index
>> instance, this ServiceContext is accessible.
>>
>> So, my final question is: is it OK to access AppConfig from Hyracks code
>> by specifying the exact parameter (e.g., compiler.textsearchmemory) to get
>> some information? Technically, it's doable since the code exists in
>> Hyracks. However, most similar parameters are processed in Asterix
>> codebase, not in  Hyracks codebase. If not, what could be a way to ensure
>> this?
>>
>> Thanks for reading!
>>
>> Best,
>> Taewoo
>>

Re: Is it OK for Hyracks to access the config option?

[Taewoo Kim <wa...@gmail.com>]

Never mind about the first question. The problem becomes simpler (but still
couldn't find the solution) after talking to Chen Luo. Thanks.

Let's say the inverted index search module Blah1 (Actually SearchResult and
FixedSizeElementInvertedListCursor in case you are interested in). For this
Blah1 module to generate intermediate and final result work properly, the
caller needs to pass three parameters (frameSize, how to
allocate/deallocate buffers, and the maximum number of frames that Blah1
can have). Blah1 can be called in two ways.

1) Search using HyracksTaskCotext (class name: Task)
2) Search initiated by merge (class name: AsynchronousScheduler)

1) and 2) are independent processes so it looks like hard to associate 1)
in 2) or 2) in 1). So, for Blah1, it needs to consider 1) as default and 2)
as a special case or vice versa. Any suggestions? Again, for B+ tree, R
tree, this problem doesn't exist since there is no intermediate result
calculation.



Best,
Taewoo

On Tue, Oct 24, 2017 at 9:21 AM, Taewoo Kim <wa...@gmail.com> wrote:

> Hello All,
>
> Here is the background. I'm currently fixing an inverted-index search OOM
> issue. This could happen because the core inverted-index search module does
> not have any budget limit. One might ask that B+-tree and R-tree search
> doesn't have any budget limit either. However, a difference is that an
> inverted-index has an inverted list (list of primary keys per secondary
> key). That is, for a single PK, there can be multiple secondary keys. So,
> dealing with this requires additional frames. So, let's assume that we need
> to have multiple frames to answer a text query that utilizes an inverted
> index.
>
> Here is my issue. We will add one more parameter to the config (e.g.,
> compiler.textsearchmemory = 32MB) to control this behavior. The
> index-search core module needs to receive and conform to this budget
> information. For inverted-index search query case, it's relatively easy. I
> can use HyracksTaskContext to pass that since there's a pipeline from
> compiler to the search module (IndexSearchOperatorNodePushable). However,
> I found an issue regarding the merge case. Merge of an LSM index happens in
> Hyracks codebase and index merge code doesn't know about
> HyracksTaskContext. So, I talked to Abdullah and he shared a hint about
> ServiceContext. In the ServiceContext, there is AppConfig interface and
> this contains all config information. And when creating an LSM index
> instance, this ServiceContext is accessible.
>
> So, my final question is: is it OK to access AppConfig from Hyracks code
> by specifying the exact parameter (e.g., compiler.textsearchmemory) to get
> some information? Technically, it's doable since the code exists in
> Hyracks. However, most similar parameters are processed in Asterix
> codebase, not in  Hyracks codebase. If not, what could be a way to ensure
> this?
>
> Thanks for reading!
>
> Best,
> Taewoo
>