[DISCUSS] Compression of secondary indexes

[Ian Maxon <im...@apache.org>]

Hello fellow devs,
I wanted to bring to the list a very good question Ritik raised in a
discussion we had in the ASF Slack regarding his fix for
https://issues.apache.org/jira/browse/ASTERIXDB-3813 . Basically the
question was: Why aren't secondary indexes compressed?
I couldn't remember why, and after informally raising this question
with some other PMC members who were familiar with our storage
compression implementation, the consensus was that at the time it was
just asserted that it wouldn't be worth it. The secondary was asserted
to be very small anyway.

However a lot of progress has been made in the project since then, and
in particular we have different kinds of secondary indexes. In
particular, we have VTree and Array indexes. These can be very large
in comparison to a traditional BTree secondary index.

Given this, it kind of seems like it might be time to at least look
into if compression in certain cases might be worth it for
secondaries. What does everyone else thnik?

- Ian

Re: [DISCUSS] Compression of secondary indexes

[Mike Carey <dt...@gmail.com>]

We definitely should look, especially for VTREE indexes!  Also interesting
for index only use cases.

On Mon, Aug 31, 2026 at 8:28 AM Ian Maxon <im...@apache.org> wrote:

> Hello fellow devs,
> I wanted to bring to the list a very good question Ritik raised in a
> discussion we had in the ASF Slack regarding his fix for
> https://issues.apache.org/jira/browse/ASTERIXDB-3813 . Basically the
> question was: Why aren't secondary indexes compressed?
> I couldn't remember why, and after informally raising this question
> with some other PMC members who were familiar with our storage
> compression implementation, the consensus was that at the time it was
> just asserted that it wouldn't be worth it. The secondary was asserted
> to be very small anyway.
>
> However a lot of progress has been made in the project since then, and
> in particular we have different kinds of secondary indexes. In
> particular, we have VTree and Array indexes. These can be very large
> in comparison to a traditional BTree secondary index.
>
> Given this, it kind of seems like it might be time to at least look
> into if compression in certain cases might be worth it for
> secondaries. What does everyone else thnik?
>
> - Ian
>

Re: [DISCUSS] Compression of secondary indexes

[Ian Maxon <im...@apache.org>]

I saw Ritik posted this patch a while earlier, so I think there has
been at least some cursory investigation into it. If the claudese is
to be believed, then it's a pretty clear winner:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21625


On Fri, Sep 4, 2026 at 3:44 PM Mike Carey <dt...@gmail.com> wrote:
>
> I commented earlier - I like this idea - it would be cool to see some example numbers shared on the list here if/as this is being looked into.  (E.g., what are the before/after index sizes for a VTREE index on the Movie dataset that is used as the main example in VTREE docs.)
>
> On 8/31/26 8:27 AM, Ian Maxon wrote:
> > Hello fellow devs,
> > I wanted to bring to the list a very good question Ritik raised in a
> > discussion we had in the ASF Slack regarding his fix for
> > https://issues.apache.org/jira/browse/ASTERIXDB-3813 . Basically the
> > question was: Why aren't secondary indexes compressed?
> > I couldn't remember why, and after informally raising this question
> > with some other PMC members who were familiar with our storage
> > compression implementation, the consensus was that at the time it was
> > just asserted that it wouldn't be worth it. The secondary was asserted
> > to be very small anyway.
> >
> > However a lot of progress has been made in the project since then, and
> > in particular we have different kinds of secondary indexes. In
> > particular, we have VTree and Array indexes. These can be very large
> > in comparison to a traditional BTree secondary index.
> >
> > Given this, it kind of seems like it might be time to at least look
> > into if compression in certain cases might be worth it for
> > secondaries. What does everyone else thnik?
> >
> > - Ian
>

Re: [DISCUSS] Compression of secondary indexes

[Mike Carey <dt...@gmail.com>]

I commented earlier - I like this idea - it would be cool to see some example numbers shared on the list here if/as this is being looked into.  (E.g., what are the before/after index sizes for a VTREE index on the Movie dataset that is used as the main example in VTREE docs.)

On 8/31/26 8:27 AM, Ian Maxon wrote:
> Hello fellow devs,
> I wanted to bring to the list a very good question Ritik raised in a
> discussion we had in the ASF Slack regarding his fix for
> https://issues.apache.org/jira/browse/ASTERIXDB-3813 . Basically the
> question was: Why aren't secondary indexes compressed?
> I couldn't remember why, and after informally raising this question
> with some other PMC members who were familiar with our storage
> compression implementation, the consensus was that at the time it was
> just asserted that it wouldn't be worth it. The secondary was asserted
> to be very small anyway.
> 
> However a lot of progress has been made in the project since then, and
> in particular we have different kinds of secondary indexes. In
> particular, we have VTree and Array indexes. These can be very large
> in comparison to a traditional BTree secondary index.
> 
> Given this, it kind of seems like it might be time to at least look
> into if compression in certain cases might be worth it for
> secondaries. What does everyone else thnik?
> 
> - Ian