[jira] [Created] (FLINK-40911) Variant builder mishandles a nested variant and a value of exactly 16 MiB

["Ramin Gharib (Jira)" <ji...@apache.org>]

Ramin Gharib created FLINK-40911:
------------------------------------

             Summary: Variant builder mishandles a nested variant and a value of exactly 16 MiB
                 Key: FLINK-40911
                 URL: https://issues.apache.org/jira/browse/FLINK-40911
             Project: Flink
          Issue Type: Bug
          Components: API / Core
            Reporter: Ramin Gharib
            Assignee: Ramin Gharib


h3. Problem 1: embedding a field or element of another variant

{\{BinaryVariantInternalBuilder.appendVariant}} reads the value through \{{getValue()}} but passes the original \{{getPos()}}. A field or element of another variant, from \{{getField}} or \{{getElement}}, shares its parent's buffer and starts at \{{pos > 0}}. \{{getValue()}} copies that slice to a new array that starts at 0, and the builder then reads the copy at the old position. Depending on the offsets, the result is a \{{MALFORMED_VARIANT}} error or a silently wrong value.

The object and array builders of the public \{{VariantBuilder}} call this method, so user code hits it too.

{code:java}
BinaryVariant source =
        BinaryVariantInternalBuilder.parseJson(
                "\{\"a\":[7,8,9],\"b\":\"hello\",\"c\":{\"d\":true}}", false);
VariantBuilder builder = Variant.newBuilder();

// Expected [[7,8,9],"hello",\{"d":true}]. Fails with MALFORMED_VARIANT instead.
builder.array()
        .add(source.getField("a"))
        .add(source.getField("b"))
        .add(source.getField("c"))
        .build();
{code}

h3. Problem 2: a value of exactly 16 MiB

The builder asserts that every size, offset and id fits into 3 bytes. An array or object whose content fills exactly 16 MiB breaks that assertion before the size check runs. With assertions enabled, it fails with a bare \{{AssertionError}}. Without them, it fails with \{{VARIANT_SIZE_LIMIT_EXCEPTION}} as intended.

{code:java}
BinaryVariantInternalBuilder builder = new BinaryVariantInternalBuilder(false);
// With its 5-byte header, this binary value fills exactly 16 MiB.
builder.appendBinary(new byte[BinaryVariantUtil.SIZE_LIMIT - 5]);
// AssertionError with -ea, VARIANT_SIZE_LIMIT_EXCEPTION without.
builder.finishWritingArray(0, new ArrayList<>(List.of(0)));
{code}

h3. Fix

* Read the shared buffer at the variant's own position: \{{appendVariantImpl(v.rawValue(), v.getMetadata(), v.getPos())}}. This also saves the copy.
* Throw \{{VARIANT_SIZE_LIMIT_EXCEPTION}} instead of asserting when a size, offset or id does not fit into 3 bytes. Such a value is always over the size limit.

Both bugs are in every release with VARIANT, 2.1.0 and later. The fix is the first commit of https://github.com/apache/flink/pull/29369 and can be cherry-picked on its own.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)