You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@resilientdb.apache.org by Harish Gokul <kh...@ucdavis.edu.INVALID> on 2025/08/21 16:47:46 UTC

[DISCUSS] Proposal: Composite Key Support in ResilientDB Storage Layer

Hi all,

I’d like to open a discussion on adding *composite key support* in the
ResilientDB storage layer to improve secondary attribute indexing.

Currently, applications that need secondary lookups either:

   -

   Fetch all values and filter in memory, or
   -

   Export and sync data to an external datastore.

This creates inefficiencies and complexity for applications using
ResilientDB as a KV or document store (e.g., ResLens, ResCanvas,
Coinsensus).

Composite keys provide a lightweight solution, enabling efficient secondary
lookups directly in the storage layer, similar to MyRocks (MySQL) and
Hyperledger Fabric.

We’ve drafted a detailed proposal (background, problem statement, solution,
trade-offs, and API changes) in the following GitHub issue:
GitHub Issue: https://github.com/apache/incubator-resilientdb/issues/180

Please share your thoughts on this approach. If the community agrees, we
can move forward with a design doc and implementation plan.

Best,
Harish Krishnakumar
PPMC , Apache ResilientDB

Re: [DISCUSS] Proposal: Composite Key Support in ResilientDB Storage Layer

Posted by Harish Gokul <ha...@gmail.com>.
Hi all,

Here is a brief design documentation for the CompositeKey Proposal with
some preliminary results.
https://github.com/apache/incubator-resilientdb/issues/180

Hope to discuss this further in great detail.

Best,
Harish Krishnakumar
PPMC, Apache ResilientDB

On Fri, Aug 22, 2025 at 2:19 PM Harish Gokul <ha...@gmail.com>
wrote:

> Hi all,
>
> design doc is a WIP. Please find the presentations below for a high-level
> introduction to the problem statement.
>
>  -
> https://docs.google.com/presentation/d/17yUOYqqTqB1eja_va5qSCEIc8B62wU6Xa5qbB3NJT9k/edit?usp=sharing
>  -
> https://docs.google.com/presentation/d/1PZd-7EtDAsY8WcHO6KkyVGfVoeThAWbtvaEeLX-eRmw/edit?usp=sharing
>
> On Fri, Aug 22, 2025 at 2:10 PM Junchao Chen <ju...@ucdavis.edu.invalid>
> wrote:
>
>> Where is the detailed proposal (including the design)?
>>
>>
>>
>> On Fri, Aug 22, 2025 at 12:48 AM Harish Gokul <khgokul@ucdavis.edu.invalid
>> >
>> wrote:
>>
>> > Hi all,
>> >
>> > I’d like to open a discussion on adding *composite key support* in the
>> > ResilientDB storage layer to improve secondary attribute indexing.
>> >
>> > Currently, applications that need secondary lookups either:
>> >
>> >    -
>> >
>> >    Fetch all values and filter in memory, or
>> >    -
>> >
>> >    Export and sync data to an external datastore.
>> >
>> > This creates inefficiencies and complexity for applications using
>> > ResilientDB as a KV or document store (e.g., ResLens, ResCanvas,
>> > Coinsensus).
>> >
>> > Composite keys provide a lightweight solution, enabling efficient
>> secondary
>> > lookups directly in the storage layer, similar to MyRocks (MySQL) and
>> > Hyperledger Fabric.
>> >
>> > We’ve drafted a detailed proposal (background, problem statement,
>> solution,
>> > trade-offs, and API changes) in the following GitHub issue:
>> > GitHub Issue:
>> https://github.com/apache/incubator-resilientdb/issues/180
>> >
>> > Please share your thoughts on this approach. If the community agrees, we
>> > can move forward with a design doc and implementation plan.
>> >
>> > Best,
>> > Harish Krishnakumar
>> > PPMC , Apache ResilientDB
>> >
>>
>

Re: [DISCUSS] Proposal: Composite Key Support in ResilientDB Storage Layer

Posted by Harish Gokul <ha...@gmail.com>.
Hi all,

design doc is a WIP. Please find the presentations below for a high-level
introduction to the problem statement.

 -
https://docs.google.com/presentation/d/17yUOYqqTqB1eja_va5qSCEIc8B62wU6Xa5qbB3NJT9k/edit?usp=sharing
 -
https://docs.google.com/presentation/d/1PZd-7EtDAsY8WcHO6KkyVGfVoeThAWbtvaEeLX-eRmw/edit?usp=sharing

On Fri, Aug 22, 2025 at 2:10 PM Junchao Chen <ju...@ucdavis.edu.invalid>
wrote:

> Where is the detailed proposal (including the design)?
>
>
>
> On Fri, Aug 22, 2025 at 12:48 AM Harish Gokul <khgokul@ucdavis.edu.invalid
> >
> wrote:
>
> > Hi all,
> >
> > I’d like to open a discussion on adding *composite key support* in the
> > ResilientDB storage layer to improve secondary attribute indexing.
> >
> > Currently, applications that need secondary lookups either:
> >
> >    -
> >
> >    Fetch all values and filter in memory, or
> >    -
> >
> >    Export and sync data to an external datastore.
> >
> > This creates inefficiencies and complexity for applications using
> > ResilientDB as a KV or document store (e.g., ResLens, ResCanvas,
> > Coinsensus).
> >
> > Composite keys provide a lightweight solution, enabling efficient
> secondary
> > lookups directly in the storage layer, similar to MyRocks (MySQL) and
> > Hyperledger Fabric.
> >
> > We’ve drafted a detailed proposal (background, problem statement,
> solution,
> > trade-offs, and API changes) in the following GitHub issue:
> > GitHub Issue: https://github.com/apache/incubator-resilientdb/issues/180
> >
> > Please share your thoughts on this approach. If the community agrees, we
> > can move forward with a design doc and implementation plan.
> >
> > Best,
> > Harish Krishnakumar
> > PPMC , Apache ResilientDB
> >
>

Re: [DISCUSS] Proposal: Composite Key Support in ResilientDB Storage Layer

Posted by Junchao Chen <ju...@ucdavis.edu.INVALID>.
Where is the detailed proposal (including the design)?



On Fri, Aug 22, 2025 at 12:48 AM Harish Gokul <kh...@ucdavis.edu.invalid>
wrote:

> Hi all,
>
> I’d like to open a discussion on adding *composite key support* in the
> ResilientDB storage layer to improve secondary attribute indexing.
>
> Currently, applications that need secondary lookups either:
>
>    -
>
>    Fetch all values and filter in memory, or
>    -
>
>    Export and sync data to an external datastore.
>
> This creates inefficiencies and complexity for applications using
> ResilientDB as a KV or document store (e.g., ResLens, ResCanvas,
> Coinsensus).
>
> Composite keys provide a lightweight solution, enabling efficient secondary
> lookups directly in the storage layer, similar to MyRocks (MySQL) and
> Hyperledger Fabric.
>
> We’ve drafted a detailed proposal (background, problem statement, solution,
> trade-offs, and API changes) in the following GitHub issue:
> GitHub Issue: https://github.com/apache/incubator-resilientdb/issues/180
>
> Please share your thoughts on this approach. If the community agrees, we
> can move forward with a design doc and implementation plan.
>
> Best,
> Harish Krishnakumar
> PPMC , Apache ResilientDB
>