You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "raghav-reglobe (via GitHub)" <gi...@apache.org> on 2026/10/17 16:30:35 UTC

[PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

raghav-reglobe opened a new pull request, #69222:
URL: https://github.com/apache/doris/pull/69222

   ### What problem does this PR solve?
   
   Issue Number: close #69220
   
   Related PR: none
   
   Problem Summary:
   
   This PR implements the proposal in #69220 as a suggestion for how the V2 path can keep an exact number on a declared DECIMAL template path. The alternative discussed there, an ondemand-based collector that keeps the raw token, is open: if the maintainers prefer that direction I will rework this PR to it.
   
   With a Schema Template that declares a DECIMAL path, a JSON number on that path was parsed to a binary64 before the
   template was applied, so digits beyond what a double carries were lost (`0.123456789012345678` stored as
   `0.123456789012345680`; the regression suite pins `188118222.011121933` → `188118222.011121920`), while the same value
   as a JSON string was exact. The parser is schema-blind by design; this change keeps it that way and carries the
   template's DECIMAL paths to the parse site:
   
   - `TScalarType.variant_decimal_paths` (optional): the DECIMAL / ARRAY<DECIMAL> predefined fields of a VARIANT type,
     filled by `VariantType.toThrift`. An old BE ignores the field; a BE that receives none behaves as before.
   - `DataTypeVariantV2` carries the paths as an ingestion hint (never part of the type's identity) and hands them to its
     serde; `parse_to_variant` / `try_parse_to_variant` read them from their result type.
   - Before parsing, one pass over the text quotes every number token with a fraction and no exponent that sits on a
     declared DECIMAL path. The unchanged parser then takes the string route it already has and the shredder casts the
     text to the template's DECIMAL exactly. Integers, exponent forms and undeclared paths are untouched (inferred types
     do not move); overflow still yields NULL; malformed text falls through to the parser's own verdict; rows without such
     a token are parsed from the original buffer.
   - `INSERT … SELECT`: the sink casts the slot that carries the parse call; the projects merge in the rewrite phase, where
     `SimplifyCastRule` rewrites `CAST(parse_to_variant(x) AS variant<template>)` into the call typed with the template.
     Stream Load already types the call with the column's VARIANT type.
   
   ### Release note
   
   A JSON number on a DECIMAL path of a VARIANT Schema Template is now stored exactly (previously it went through a double
   first; only the string form was exact).
   
   ### Check List (For Author)
   
   - Test
       - [x] Unit Test — BE: `VariantDecimalPathsTest` (pre-pass + path set), `DataTypeVariantV2SerdeInputTest`,
         `FunctionVariantParseTest`; FE: `VariantTypeDecimalPathsThriftTest`, `SimplifyCastRuleTest`.
       - [x] Regression test — the pinned lossy rendering `188118222.011121920` in `variant_p0/predefine` and
         `variant_p0/v2/predefine` (load.out + sql/q02.out, 42 lines) becomes the exact `188118222.011121933` (updated in
         this PR).
       - [x] Manual test — Stream Load JSON and CSV, and `INSERT … SELECT parse_to_variant(text)` in plain, UNION ALL, CTE
         and LIMIT-subquery shapes, on a two-FE / seven-BE cluster: number form exact on declared DECIMAL paths, string form
         unchanged, undeclared paths still DOUBLE.
   
   - Behavior changed:
       - [x] Yes. The number form on a declared DECIMAL template path keeps all its digits; the FAQ paragraph that
         recommends the string form becomes advice rather than a requirement.
   
   - Does this need documentation?
       - [x] Yes. The FAQ entry on DECIMAL template paths (doris-website) — follow-up PR once this merges.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Read a JSON number on a DECIMAL or LARGEINT Schema Template path from its text [doris]

Posted by "lsy3993 (via GitHub) via commits" <co...@doris.apache.org>.
lsy3993 commented on PR #69222:
URL: https://github.com/apache/doris/pull/69222#issuecomment-6912524434

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69222:
URL: https://github.com/apache/doris/pull/69222#issuecomment-6236674886

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Read a JSON number on a DECIMAL or LARGEINT Schema Template path from its text [doris]

Posted by "raghav-reglobe (via GitHub) via commits" <co...@doris.apache.org>.
raghav-reglobe commented on code in PR #69222:
URL: https://github.com/apache/doris/pull/69222#discussion_r4561345828


##########
be/src/core/data_type_serde/data_type_variant_v2_serde.cpp:
##########
@@ -643,10 +645,16 @@ JsonToVariantOptions serde_json_options() {
 
 } // namespace
 
+JsonToVariantOptions DataTypeVariantV2SerDe::json_options() const {
+    JsonToVariantOptions options = serde_json_options();
+    options.template_paths = _template_paths;

Review Comment:
   @eldenmoon the finding is correct, and I would like your call on the fix before I build it, because it is a choice about aggregate state and not about this PR's parsing.
   
   The state of a generic `collect_list` is JSON text per element (`aggregate_function_collect.h`, `write` / `read`), read back with the argument type's serde. JSON text cannot say whether `1.005` was a DOUBLE or a DECIMAL, so the serde has to pick one way to read it:
   
   1. **Read the state without the template** (drop it from `TAggregateExpr.param_types`, as the CAST node already does). The case in this thread then gives 1.00 in every phase. The cost: a value this PR stores exactly on a templated column, for example `0.123456789012345678` on `decimal(38,18)` or `18446744073709551616` on a LARGEINT path, comes back as a DOUBLE after a two-phase `collect_list`. The suite pins those two values today (`collect_one_phase` / `collect_two_phases`), and I expect the two-phase result to change.
   2. **Keep the value's own encoding in the state** (write the element in the column's binary form instead of JSON text). Both cases come back as they went in. It changes the state format of `collect_list` / `array_agg` over a VARIANT argument, so it needs a `be_exec_version` step, or your word that VARIANT V2 does not need one yet.
   
   I think 2 is the correct one, but it is the larger change and it touches aggregate state, which this PR did not set out to change. If you prefer, I do 1 here, state the limit in the description, and send 2 as its own PR.
   
   P2 (the padded buffer outside memory accounting) is a plain fix and comes with the next push.
   
   Still open from my earlier reply: whether you want the INSERT / UPDATE / MERGE typing rule in this PR or as a follow-up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #69222:
URL: https://github.com/apache/doris/pull/69222#issuecomment-6263052252

   Thanks — but I read this as the documented contract rather than a defect:
   
   > “Even declaring the subpath as DECIMAL via the Schema Template … does not fully guarantee precision, because the value is first parsed as DOUBLE and then converted to DECIMAL on the write path. If the JSON value is written as a string … the string is parsed directly into DECIMAL on write, preserving precision.”
   
   
   The Schema Template is documented as a storage-side declaration: it guides the subcolumn type, and precision for the JSON number form is bounded by binary64. The supported way to keep full precision is to send the value as a quoted string, and users who need exact DECIMAL already have that path. The PR body itself says the FAQ will have to be rewritten, and “Behavior changed: Yes” — so this is a change of a documented contract, not a bug fix, and it should be justified as such.
   
   I don’t think this patch can carry that change yet:
   
   It does not build. variant_util::glob_match_re2 (be/src/core/value/variant/variant_decimal_paths.cpp:48) is neither declared nor defined at this head: git grep glob_match_re2 returns only that call site, and variant_util.h has no such declaration.
   It only covers -?digits.digits. Integers beyond 2^64 and exponent forms still go through double, so the new contract would read “declaring DECIMAL guarantees precision, except for some number syntaxes” — much harder to document than today’s clean rule.
   It leaks the template into query results. The hint is attached by a global cast rule (SimplifyCastRule), while CAST(variant AS variant) is an identity in BE and only the shredder converts. So SELECT CAST(parse_to_variant(s) AS VARIANT<a:decimal>) now returns a JSON string on that path.
   It adds a full document copy + byte-wise scan per row on the ingestion path as soon as any DECIMAL path is declared.
   If the team does want to change the contract, the clean direction is to keep the number text available to the write path (ondemand raw_json_token, as discussed in #69220), or to type the parse call only in the insert sink and emit a DECIMAL — not a quoted string. Otherwise I’d keep the current documented behavior and close/rework this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

Posted by "raghav-reglobe (via GitHub) via commits" <co...@doris.apache.org>.
raghav-reglobe commented on code in PR #69222:
URL: https://github.com/apache/doris/pull/69222#discussion_r4546128178


##########
regression-test/data/variant_p0/predefine/sql/q02.out:
##########
@@ -1,6 +1,6 @@
 -- This file is automatically generated. You should know what you did if you want to edit this
 -- !q02 --

Review Comment:
   Added `sql/q03` to both `variant_p0/predefine` and `variant_p0/v2/predefine`, with the same cases in each. The tables are created at the end of `load.groovy`, so `load.out` is unchanged.
   
   Every document is loaded three ways: `INSERT … SELECT parse_to_variant`, `INSERT … SELECT try_parse_to_variant`, and Stream Load JSON. Six are also loaded through the `INSERT … VALUES` fast path: 38 fraction digits on decimal(38,38), 1.005 on decimal(10,2), and 2^64, 2^127−1, −2^127 and 2^127 on LARGEINT. Each is read back whole, by typed subcolumn, and as text.
   
   - **DECIMAL**
     - decimal(38,0) and decimal(38,38) keep full precision.
     - On decimal(10,2), extra fraction digits are rounded from the exact text: 1.005 → 1.01 and 0.125 → 0.13.
     - 1.5e3 → 1500.00, 15E-1 → 1.50, −0.0 → 0.00, −1.50 stays −1.50.
     - 43 significant digits on decimal(38,38) are rounded correctly.
     - Overflows give NULL: a 39-digit integer on decimal(38,0), 123456789.123 on decimal(10,2), and 1e400.
     - Also covered: ARRAY<DECIMAL>, a glob field before a named one and the reverse (the first match wins), and a declared path under an array element, which stays off the path.
   - **Big integer on a LARGEINT path**
     - Exact: 2^63−1, 2^63, 2^64−1, 2^64, −2^63−1, 10^38−1, 10^38, ±(2^127−1) and −2^127.
     - 2^127 gives NULL.
     - 1.5 → 1, as before.
     - Also covered: the string form, ARRAY<LARGEINT>, a nested path, a glob field, and one load that mixes a value below 10^38 with one above it.
     - One array mixing an int64 with 2^127−1 reads back exact, `[1, 170141183460469231731687303715884105727]`. 2^127−1 as a number and as a JSON string, loaded together, read back alike by subcolumn, as text and in the document.
   - **BIGINT path:** 2^63 gives NULL through the cast.
   - **Off any template path:** 2^63 loads. 2^64 and 2^127−1 keep today's behaviour: `parse_to_variant` fails, `try_parse_to_variant` and Stream Load store a NULL variant (the Stream Load runs with `max_filter_ratio` 0).
   - **Doc mode** (`variant_p0/doc_mode/predefine/test_predefine_largeint_text`, new): doc mode accepts a LARGEINT template field, so the text read reaches it. Ten documents, three load ways, one table reading paths from the document and one with materialized subcolumns. The document column keeps each value as parsed, before the template's cast: up to 10^38−1 an unquoted number, above that the quoted digits, the same as the string form. Every LARGEINT reads back exact by subcolumn, on both tables. 2^127 keeps its digits in the document and reads back NULL as a LARGEINT; as a string it gives the digits where the path is read from the document, and NULL where the subcolumn is materialized.
   - **Other expressions and statements** (`variant_p0/predefine/test_predefine_template_expr_and_merge`, new), on a template with decimal(10,2), decimal(20,9) and LARGEINT paths: the call under `if`, `case when` (searched and simple), `coalesce`, `nvl` and `nullable`; an `if` over a subquery's column; `MERGE INTO`, both branches; `UPDATE`; and a table with `variant_enable_typed_paths_to_sparse`. Each reads back 1.005 → 1.01 and 0.125 → 0.13, 188118222.011121933 whole, and 2^64 and 2^127−1 exact.
   
   I also ran every suite under `variant_p0/predefine`, `variant_p0/v2/predefine` and `variant_p0/doc_mode/predefine`, and 20 other `variant_p0` suites over typed templates (types, hierarchical, delete/update, rqg4/rqg7, compute_v2, topn, load_invalid_json, parse_functions, …), on a single-FE/single-BE build of this branch. All pass, with these notes:
   - `test_variant_multi_index_nonCurrent`, `test_variant_predefine_types_with_indexes_profile` and `test_variant_is_null_expr` need `enable_debug_points`, which my build turns on.
   - `doc_mode/test_outfile_csv_variant_type` needs S3 credentials my build doesn't have, so it did not run to completion.
   
   Two tests were wrong at the previous head; both are fixed:
   - `TypeVariantTemplateParseTest`: its source table is empty, so `PRUNE_EMPTY_PARTITION` replaced the scan with an empty relation, and the projection carrying the call disappeared. The class now disables that rule, as the other planner tests over empty tables do (4a7f46cb134).
   - `q02_13` (`decimal_ >= 188118222.011121933`): row 1 holds exactly that number and is now returned. The expected output still had the row set from when the value went through a double (6a68fc45fae).
   
   Also in this push:
   - **A bug in the DECIMAL text reader**, found by a local review. It stopped adding exponent digits at 1000, so an exponent of five or more digits was cut short. With a long enough fraction the value read was wrong: `0.<999 zeros>1e10000`, which is 10^9000, was read as 1. Fixed in 92d1b34eb69: every exponent is now read in full until it can no longer change the result. I also compared both text readers with an exact reference on 300,000 generated tokens; they agree on all of them.
   - **`MERGE INTO`, and a call under `if`, `case when`, `coalesce` or `nvl`**, were not typed, so they kept the read through a double and still failed on an integer beyond 64 bits. They are typed now (34736543c9f). `MERGE INTO` wraps every value it assigns in an `if` of its own, which is why it was missed.
   - **The call is typed where it is computed.** Inside a statement that loads a templated column, a filter, join or sort over the parsed value therefore sees what the template reads: `cast(v['amount'] as string)` is `1.50` there and `1.5` in a plain SELECT. The new suite pins this. Say if you would rather keep such readers on the plain read; that needs a second call for the sink.
   - **Under a template without a DECIMAL field the DOM parser now reads first** (bee6f544dab), so a table whose template declares only LARGEINT fields pays the text read only for a document the DOM parser rejects. A document holding an integer beyond 64 bits now costs a failed DOM parse plus the text read; I have not measured either side. The big-integer table of `q03` has such a template, and its expected output is unchanged.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

Posted by "raghav-reglobe (via GitHub) via commits" <co...@doris.apache.org>.
raghav-reglobe commented on code in PR #69222:
URL: https://github.com/apache/doris/pull/69222#discussion_r4546125865


##########
gensrc/thrift/Types.thrift:
##########
@@ -136,6 +136,15 @@ enum TInvertedIndexFileStorageFormat {
     SNII = 4     // SNII native inverted index storage format
 }
 
+// One field of a VARIANT Schema Template, in declaration order, for the BE's JSON parser: a number on a
+// path whose first matching field is DECIMAL (or ARRAY<DECIMAL>) is parsed from its text instead of
+// through a binary64. The whole template rides so the BE matches a path the way the shredder does.
+struct TVariantTemplatePath {
+  1: required string pattern
+  2: required bool is_glob
+  3: required bool is_decimal

Review Comment:
   Done. `TVariantTemplatePath.type` is now the field's `TPrimitiveType` (an ARRAY field sends its item type) and `is_decimal` is gone. The BE decides how a number on a path is read from that type, matching the first field in declaration order as before:
   - DECIMAL32/64/128I/256: the number is read from its token text into an exact decimal, as in the previous revision (this push fixes that reader's handling of a long exponent; see the other thread).
   - LARGEINT: an integer simdjson cannot hold in 64 bits is now read from its text into an int128. Before this, such a document left the typed parser and failed. Up to 10^38−1 it is stored as a scale-0 decimal; above that it is stored as its digit text. Either way the cast to LARGEINT is exact, and a value outside the LARGEINT range ends up NULL through the cast. In a regular (non-doc-mode) VARIANT a number now ends up the same as its JSON-string form: same subcolumn value, same document text, which prints a LARGEINT longer than 38 digits quoted for the string form as well. In doc mode the document column keeps the value as parsed, as it does for every typed path.
   - Any other type: the native read, as before.
   
   The template is sent when it declares a DECIMAL or LARGEINT field, or an ARRAY of one; otherwise nothing is sent, as before. With a DECIMAL field every document is read with the ondemand parser, because the DOM parser turns a fraction into a double without an error and leaves nothing to fall back on. With LARGEINT fields only, the DOM parser reads the document first, as without a template, and every document it rejects is read again from its text; a document holding an integer beyond 64 bits is therefore parsed twice.
   
   I sent the primitive type rather than a full `TTypeDesc`. The struct sits inside `TScalarType`, so a `TTypeDesc` there would make the definition recursive. Precision and scale are applied by the cast to the declared type, so the parser only needs the kind.
   
   The struct is new in this PR, so changing field 3 from bool to an enum affects no release.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Read a JSON number on a DECIMAL or LARGEINT Schema Template path from its text [doris]

Posted by "eldenmoon (via GitHub) via commits" <co...@doris.apache.org>.
eldenmoon commented on PR #69222:
URL: https://github.com/apache/doris/pull/69222#issuecomment-6841177267

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

Posted by "raghav-reglobe (via GitHub) via commits" <co...@doris.apache.org>.
raghav-reglobe commented on PR #69222:
URL: https://github.com/apache/doris/pull/69222#issuecomment-6657018044

   <!-- doris-repo-review:v1:begin -->
   ### Local pipeline review — ✅ PASS
   
   ```yaml
   schema: doris-repo-review/v1
   status: PASS
   pr: apache/doris#69222
   commit: 4e2bcb1d60f0ccb0f4b1b6c65afdd735adf45305
   base: 9cb2d8a5e9a96b65647107a1e9bfd9237e860660
   reviewed_at: 2026-11-05T17:08+05:30
   reviewer: raghav-reglobe
   model: claude-fable-5-1
   effort: xhigh
   findings: {blocker: 0, major: 0, minor: 0, nit: 0}
   rounds: 2
   converged: true
   ```
   
   **Notes for maintainers**
   
   - Reviewed as the unpushed candidate (15 commits on master 1503e390275c, rebased) right before this force-with-lease push; the two earlier local runs of this PR (head d083daf: REQUEST_CHANGES 1/1/13/4; candidate 4e84ad6: REQUEST_CHANGES 1/0/2/1) were inherited and every one of their findings is fixed at this head by content.
   - Two design choices for the variant owners: (1) the typed `parse_to_variant` call is produced by a sink-walk rewriter for every INSERT shape (VALUES, SELECT, CTE, UNION, with and without a column list) instead of bind-time typing; (2) the thrift carrier is `TVariantTemplatePath {pattern, is_glob, is_decimal}` (renamed from `TVariantDecimalPath`, precision/scale dropped — the BE resolves them from the column's template); the struct is unreleased on every release branch (TScalarType's last field id is 7 on master/4.2/4.1 and 5 on 4.0/3.1).
   - This push's round: `withReturnType` builds the typed call outside `keepFunctionSignature(true)` so the call's signature — and the template on the wire — reaches the BE on every rewrite path; an array nested inside an array stays off every declared path; the test pins `call.getDataType()`; clang-format 16 on four test statements.
   - Not verified locally: no build and no test run (no JDK or BE toolchain on the review machine); compile correctness and the test expectations were walked by reading; the four `.out` files are the PR's own.
   
   <sub>Reviewed locally with the `doris-repo-review` pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.</sub>
   <!-- doris-repo-review:v1:end -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Read a JSON number on a DECIMAL or LARGEINT Schema Template path from its text [doris]

Posted by "github-actions[bot] (via GitHub) via commits" <co...@doris.apache.org>.
github-actions[bot] commented on code in PR #69222:
URL: https://github.com/apache/doris/pull/69222#discussion_r4558762446


##########
be/src/core/data_type_serde/data_type_variant_v2_serde.cpp:
##########
@@ -643,10 +645,16 @@ JsonToVariantOptions serde_json_options() {
 
 } // namespace
 
+JsonToVariantOptions DataTypeVariantV2SerDe::json_options() const {
+    JsonToVariantOptions options = serde_json_options();
+    options.template_paths = _template_paths;

Review Comment:
   [P1] Preserve Variant values when reading aggregate intermediate state
   
   The template also reaches `collect_list` through `TAggregateExpr.param_types`, even when `dropVariantTemplates` removed it from the executed CAST. Consider this reduced ordinary SELECT plan with one source row `j = '{"a":1.005}'`:
   
   ```text
   Aggregate(collect_list(v) AS xs)
     Project(CAST(parse_to_variant(j) AS VARIANT<'a':DECIMAL(10,2)>) AS v)
       Scan(src)
   ```
   
   With one aggregation phase, the VARIANT-to-VARIANT cast preserves native DOUBLE 1.005, so `CAST(xs[1]['a'] AS DECIMAL(10,2))` is 1.00. With two phases, `ExpressionTranslator.visitAggregateExpression` sends the original templated parameter type, `AggFnEvaluator` prefers that type over the stripped CAST node's type, and `AggregateFunctionCollectListData::read` uses its serde to reparse the intermediate JSON. This assignment therefore turns that same value into exact DECIMAL 1.005 and the result becomes 1.01. The existing phase-equivalence test starts from already typed stored values and misses this case. Aggregate state must preserve the input's numeric representation rather than reapply an ingestion hint to CAST-produced values.



##########
be/src/exprs/function/parse/variant_string_parse.cpp:
##########
@@ -397,9 +685,91 @@ struct JsonStringToVariantEncoder::Impl {
         row.finish();
     }
 
+    // Whether the first byte of the document opens an object or an array, past what simdjson
+    // itself passes over before a document: a UTF-8 byte order mark and JSON whitespace.
+    static bool starts_a_container(StringRef json) {
+        size_t pos = 0;
+        if (json.size >= 3 && std::memcmp(json.data, "\xEF\xBB\xBF", 3) == 0) {
+            pos = 3;
+        }
+        for (; pos < json.size; ++pos) {
+            const char c = json.data[pos];
+            if (c != ' ' && c != '\t' && c != '\n' && c != '\r') {
+                return c == '{' || c == '[';
+            }
+        }
+        return false;
+    }
+
+    // Adds the document as the ondemand pass reads it, or nothing: false means the row it began
+    // was rolled back, and the caller gives the text the DOM parser's verdict.
+    bool add_row_with_template_paths(StringRef json) {
+        auto row = builder.begin_row();
+        bool collected = false;
+        // The DOM parser reads the whole text before any depth or key check runs, and the ondemand
+        // pass checks as it reads. So on a document whose prefix is over-deep or duplicate-keyed
+        // AND whose tail is malformed, {"o":{"a":1,"a":2},"k":x}, the ondemand pass meets the
+        // builder's check where the DOM parser meets the syntax error. It would throw where the
+        // lenient serde stored the text as a string row, and the strict functions would report
+        // the prefix's defect where they reported the syntax error. Let the DOM parser give the
+        // verdict in both: a well-formed document meets the same check there and fails as it
+        // always did.
+        try {
+            collected = collect_with_template_paths(row, json);
+        } catch (const Exception& exception) {
+            if (exception.code() != ErrorCode::INVALID_ARGUMENT) {
+                throw;
+            }
+        }
+        if (!collected) {
+            // `row` ends here and rolls its partial content back
+            return false;
+        }
+        row.finish();
+        return true;
+    }
+
+    // A root object or array read with the ondemand parser; false on any simdjson error, and for a
+    // root scalar, which has no path.
+    bool collect_with_template_paths(VariantBatchBuilder::Row& row, StringRef json) {
+        // One padded copy per encoder, grown to the largest document seen: ondemand values point
+        // into it, and the builder copies a row's bytes before the next document is read.
+        const size_t padded_size = json.size + simdjson::SIMDJSON_PADDING;
+        if (padding_buffer.size() < padded_size) {
+            padding_buffer.resize(padded_size);

Review Comment:
   [P2] Charge the retained padded JSON buffer to the memory tracker
   
   `padding_buffer` is a `std::string`, so this resize allocates an additional full-document copy without Doris allocator accounting. The encoder retains its capacity for the largest document until the whole batch ends. Every object/array under a DECIMAL template reaches this allocation, as does the LARGEINT fallback; parallel encoders each retain their own copy. The raw malloc hook only forwards to jemalloc, so neither query/load consumption nor allocation-time limit checks include these bytes. The existing third-party DOM parser buffer does not account for this newly added copy. Use a Doris allocator-backed byte buffer such as `DorisVector<char>`, or explicitly check, consume and release its allocated capacity.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

Posted by "raghav-reglobe (via GitHub) via commits" <co...@doris.apache.org>.
raghav-reglobe commented on PR #69222:
URL: https://github.com/apache/doris/pull/69222#issuecomment-6657040441

   @eldenmoon — this PR touches the Variant core, so two design choices are worth your eyes before the auto-requested FE reviewers get to the Nereids half:
   
   1. **Which call gets typed.** The template is attached by a sink-scoped rewrite (`TypeVariantTemplateParse`): from the OLAP sink down to the `parse_to_variant` / `try_parse_to_variant` call that feeds a templated column, through VALUES, SELECT, CTE, UNION and derived-table shapes. A SELECT outside a sink keeps its cast and the untyped call, and `SimplifyCastRule` is untouched. The earlier draft typed the call in `SimplifyCastRule`; that missed the shapes where the cast and the call never become adjacent.
   2. **What crosses the wire.** `TScalarType.variant_template_paths` carries the whole Schema Template in declaration order as `TVariantTemplatePath {pattern, is_glob, is_decimal}` (renamed from the draft's `TVariantDecimalPath`; precision and scale dropped — the emitted decimal's width follows the text's own digits, and the column's cast decides the rest). The BE matches a path the way the shredder does: first field in declaration order, `==` for a literal, `fnmatch(FNM_PATHNAME)` for a glob; an object or an array nested inside an array keeps its subtree off every declared path. The struct exists on no release branch, and a BE that receives no paths behaves as before.
   
   If you would rather see the template matched elsewhere (for instance the shredder's own matcher exposed to the parser), say so and I will rework it — the BE text reader and its tests stay the same either way.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

Posted by "raghav-reglobe (via GitHub)" <gi...@apache.org>.
raghav-reglobe commented on PR #69222:
URL: https://github.com/apache/doris/pull/69222#issuecomment-6288030516

   Thanks for the careful read — all four points were right, and the PR is reworked along the first direction you named: the number's text now reaches the write path.
   
   **What changed (force-pushed):**
   
   1. **The pre-pass is gone.** With DECIMAL template paths present, the document is read with simdjson's **ondemand** parser (`OndemandTreeCollector`, the twin of `JsonTreeCollector`), which has every number's `raw_json_token()`. A number on a declared path is read from its text straight into a Variant decimal (`Row::add_decimal`), so the shredder's cast is decimal → decimal. No quoted strings, no second scan, no extra copy beyond the padded buffer the parse itself needs. Documents without paths, numbers off the declared paths and root scalars go through the DOM collector exactly as before; any ondemand error falls back to the DOM parser's verdict (the row is rolled back first).
   2. **Every number syntax on a declared path is exact now:** fractions, exponent forms and integers beyond 64 bits (`parse_json_number_as_decimal` folds the exponent into the scale, ≤ 38 significant digits). Beyond 38 significant digits the text is kept as a string and the template's cast decides — a correctly rounded value or NULL — never a binary64. So the contract becomes one sentence: a JSON number on a DECIMAL template path keeps its digits.
   3. **Query results:** the typed parse call now yields a decimal holding the document's own digits on that path, which is what a cast to that templated type declares; it no longer turns a number into a JSON string. The FE hook is unchanged (it is the only place where the merged `CAST(parse_to_variant(x) AS variant<…>)` shape is visible); I'm happy to narrow it to the sink if you prefer that shape.
   4. **The matcher is `fnmatch(…, FNM_PATHNAME)`,** the same call the shredder uses for typed paths — the stray `glob_match_re2` reference was mine and is gone, and the branch is rebased onto current master.
   
   On "contract vs. defect": agreed that the FAQ documents today's behaviour; the PR body now frames this as the change it is (the FAQ paragraph becomes advice rather than a requirement, follow-up docs PR once this lands). The regression expectation change is the same single value as before (`188118222.011121920` → `…933`).
   
   BE tests: `VariantDecimalPathsTest` (matcher + the decimal text reader), `DataTypeVariantV2SerdeInputTest` (exact fractions, exponent, big integer, >38 digits kept as text, off-path and invalid-text verdicts unchanged), `FunctionVariantParseTest`; FE: `VariantTypeDecimalPathsThriftTest`, `SimplifyCastRuleTest`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Read a JSON number on a DECIMAL or LARGEINT Schema Template path from its text [doris]

Posted by "eldenmoon (via GitHub) via commits" <co...@doris.apache.org>.
eldenmoon commented on PR #69222:
URL: https://github.com/apache/doris/pull/69222#issuecomment-6841196824

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Read a JSON number on a DECIMAL or LARGEINT Schema Template path from its text [doris]

Posted by "eldenmoon (via GitHub) via commits" <co...@doris.apache.org>.
eldenmoon commented on PR #69222:
URL: https://github.com/apache/doris/pull/69222#issuecomment-6971823882

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

Posted by "eldenmoon (via GitHub) via commits" <co...@doris.apache.org>.
eldenmoon commented on code in PR #69222:
URL: https://github.com/apache/doris/pull/69222#discussion_r4522671260


##########
regression-test/data/variant_p0/predefine/sql/q02.out:
##########
@@ -1,6 +1,6 @@
 -- This file is automatically generated. You should know what you did if you want to edit this
 -- !q02 --

Review Comment:
   please add more  regression tests about decimal  and big integer in variant template



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

Posted by "eldenmoon (via GitHub) via commits" <co...@doris.apache.org>.
eldenmoon commented on code in PR #69222:
URL: https://github.com/apache/doris/pull/69222#discussion_r4522671260


##########
regression-test/data/variant_p0/predefine/sql/q02.out:
##########
@@ -1,6 +1,6 @@
 -- This file is automatically generated. You should know what you did if you want to edit this
 -- !q02 --

Review Comment:
   please add more  regression tests about decimal and big integer 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Parse a JSON number on a DECIMAL Schema Template path from its text [doris]

Posted by "eldenmoon (via GitHub) via commits" <co...@doris.apache.org>.
eldenmoon commented on code in PR #69222:
URL: https://github.com/apache/doris/pull/69222#discussion_r4522631066


##########
gensrc/thrift/Types.thrift:
##########
@@ -136,6 +136,15 @@ enum TInvertedIndexFileStorageFormat {
     SNII = 4     // SNII native inverted index storage format
 }
 
+// One field of a VARIANT Schema Template, in declaration order, for the BE's JSON parser: a number on a
+// path whose first matching field is DECIMAL (or ARRAY<DECIMAL>) is parsed from its text instead of
+// through a binary64. The whole template rides so the BE matches a path the way the shredder does.
+struct TVariantTemplatePath {
+  1: required string pattern
+  2: required bool is_glob
+  3: required bool is_decimal

Review Comment:
   how about using real type here? instead of a boolean flag?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](variant) Read a JSON number on a DECIMAL or LARGEINT Schema Template path from its text [doris]

Posted by "raghav-reglobe (via GitHub) via commits" <co...@doris.apache.org>.
raghav-reglobe commented on code in PR #69222:
URL: https://github.com/apache/doris/pull/69222#discussion_r4552475931


##########
regression-test/data/variant_p0/predefine/sql/q02.out:
##########
@@ -1,6 +1,6 @@
 -- This file is automatically generated. You should know what you did if you want to edit this
 -- !q02 --

Review Comment:
   A correction to my reply above. Its paragraph "The call is typed where it is computed" no longer describes the code.
   
   Since 7eedb15cc53 and ecc7f39278d the template applies to the value the column stores and to nothing else. In INSERT, UPDATE and MERGE INTO a `parse_to_variant` / `try_parse_to_variant` call is read with the template only when the templated column is the one reader of its value. When anything else reads that value, the call is left as it is: that reader sees what it sees in a plain SELECT (`1.5` for `1.50`), and the column stores what it stored before this PR. "Anything else" is a filter, a join or sort key, a GROUP BY, another column of the table, a generated column or a materialized view over the templated column, or a second copy of the same call in the statement.
   
   The reason is that a typed call changed more than the column. Every other reader of the value saw the new read too, and a comparison against a second parse of the same text stopped matching (`1.50` against `1.5`). A reader that held its own copy of the call kept the old read: a generated column `g AS (CAST(v['a'] AS DECIMAL(10,2)))` stored 1.00 beside `v['a']` = 1.01.
   
   A load is not affected. It parses the column once with the template, and its WHERE and derived columns read that value.
   
   `test_predefine_template_expr_and_merge` pins both sides in the tags `readers`, `merge_readers`, `generated`, `generated_key`, `mv_base`, `mv_index` and `load`. The PR description has the full rule.
   
   The rule that decides this, `TypeVariantTemplateParse`, is the largest FE part of the PR. If you would rather review it on its own, I can move it to a follow-up PR. This one would then keep the wire type, the two BE text readers and loads, and an explicit `parse_to_variant` call in a statement would read as before.



##########
gensrc/thrift/Types.thrift:
##########
@@ -136,6 +136,15 @@ enum TInvertedIndexFileStorageFormat {
     SNII = 4     // SNII native inverted index storage format
 }
 
+// One field of a VARIANT Schema Template, in declaration order, for the BE's JSON parser: a number on a
+// path whose first matching field is DECIMAL (or ARRAY<DECIMAL>) is parsed from its text instead of
+// through a binary64. The whole template rides so the BE matches a path the way the shredder does.
+struct TVariantTemplatePath {
+  1: required string pattern
+  2: required bool is_glob
+  3: required bool is_decimal

Review Comment:
   Three details of my reply above have changed since:
   
   - "With a DECIMAL field every document is read with the ondemand parser" now holds for a document that is an object or an array. A root scalar is on no path, so it is parsed as without a template (25f62d7bac8).
   - The three fields of `TVariantTemplatePath` are `optional` now (abbe5d4d81b). A BE that finds one missing in any entry leaves the whole template out, because the first matching field decides.
   - A CAST does not read by the template. The BE reads it in two places: the parse functions take it from their result type, and the VARIANT serde takes it from its type when it reads JSON text (a load, the state of `collect_list`). The FE takes it off the target of a CAST or TRY_CAST before the expression is sent (9146a62878f), so a cast of a JSON string to an ARRAY or STRUCT of a templated VARIANT reads what it read before this PR.
   
   One addition on the BE side: a nonzero number below every scale the field's type has is read as the 0 it rounds to (7750d214c68, c1a0a8520ca). That is below 10^-39 on a field of up to 38 digits, and below 10^-77 on a DECIMAL256 field, whose scale goes to 76 and which is told apart for that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org