You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@asterixdb.apache.org by mb...@apache.org on 2026/09/30 08:03:55 UTC
(asterixdb) branch master updated (84e7e4ec30 -> d126525a77)
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
from 84e7e4ec30 Merge "Null merge branch 'gerrit/morpheus' into 'master'"
new 5504d051bd [NO ISSUE][MISC] Record AI work with Generated-by
new beff298399 [ASTERIXDB-3915][EXT]: Read wide Parquet decimals
new be88b4bb26 [NO ISSUE][MISC] Advance the AWS SDK to 2.55.7 and woodstox to 7.3.0
new a72cb24523 Merge branch 'gerrit/trinity' into 'gerrit/morpheus'
new b96b4e8240 [NO ISSUE][MISC] Advance commons-lang3 to 3.21.0
new d126525a77 Merge branch 'gerrit/morpheus' into 'master'
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../external_dataset/ExternalDatasetTestUtils.java | 1 +
.../parquet/ParquetFileExampleGeneratorUtil.java | 46 +++++++
.../decimal-fixed-len-byte-array.1.ddl.sqlpp} | 7 +-
.../decimal-fixed-len-byte-array.2.query.sqlpp} | 5 +-
.../decimal-fixed-len-byte-array.2.adm | 3 +
.../runtimets/testsuite_external_dataset_s3.xml | 8 ++
.../converter/primitve/DecimalConverter.java | 5 +-
.../hdfs/parquet/ParquetDecimalToDoubleTest.java | 140 +++++++++++++++++++++
asterixdb/pom.xml | 6 +-
hyracks-fullstack/pom.xml | 4 +-
10 files changed, 213 insertions(+), 12 deletions(-)
copy asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/common/parquet/parquet-types/{temporal/temporal.1.ddl.sqlpp => decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.1.ddl.sqlpp} (92%)
copy asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/{ddl/invalid-dataverse/invalid-dataverse.001.ddl.sqlpp => external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.2.query.sqlpp} (88%)
create mode 100644 asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.2.adm
create mode 100644 asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/ParquetDecimalToDoubleTest.java
(asterixdb) 01/06: [NO ISSUE][MISC] Record AI work with Generated-by
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit 5504d051bdffcc0497c9f3730f1161cd6a538dad
Author: Michael Blow <mi...@couchbase.com>
AuthorDate: Tue Sep 29 09:01:23 2026 -0400
[NO ISSUE][MISC] Record AI work with Generated-by
- user model changes: no
- storage format changes: no
- interface changes: no
Details: CLAUDE.md here still asked for a Co-Authored-By trailer on
AI-assisted commits, while this repo records them with Generated-by, as
master's copy already says. Take master's Commit Messages section as it
stands, which also keeps Ext-ref inside the one trailer block.
Ext-ref: MB-74304
Generated-by: Claude Opus 5.5 <no...@anthropic.com>
Change-Id: I4ab6ec785eac4a0dcd7f4d719855841654ddd906
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21926
Tested-by: Michael Blow <mb...@apache.org>
Reviewed-by: Michael Blow <mb...@apache.org>
---
CLAUDE.md | 25 ++++++++++++++-----------
1 file changed, 14 insertions(+), 11 deletions(-)
diff --git a/CLAUDE.md b/CLAUDE.md
index 9591e89223..93bd78016c 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -74,24 +74,25 @@ change with **no topic**, or a different one — not added to the coordinated ch
## Commit Messages
-**Every commit message must end with the trailer:**
+**Every AI-assisted commit message must end with the trailer:**
```
-Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
+Generated-by: Claude Opus 5 <no...@anthropic.com>
```
-(naming whichever model actually did the work). This has repeatedly been missed and caught only
-after the change was pushed for review, which costs a message-only patchset.
+(naming whichever model actually did the work). This repo records AI contributions with
+`Generated-by:`, **not** `Co-Authored-By:` — a downstream extension may use the other form in its
+own repository, but not here. This has repeatedly been missed and caught only after the change was
+pushed for review, which costs a message-only patchset.
**Write it at commit time, not as a follow-up amend.** The footer must be one contiguous run of
-trailers, with the `Co-Authored-By` line adjacent to `Change-Id`:
+trailers — `Ext-ref:` included, with no blank line anywhere inside it:
```
<body>
<- exactly one blank line
Ext-ref: <downstream-issue>
-
-Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
+Generated-by: Claude Opus 5 <no...@anthropic.com>
Change-Id: I...
```
@@ -111,15 +112,17 @@ of it.
Two ways the footer gets destroyed, both of which make the `commit-msg` hook mint a **second**
`Change-Id` and silently orphan the change:
-- **A blank line between `Change-Id:` and `Co-Authored-By:`** splits the footer, so the last
- paragraph is read as body text.
+- **A blank line anywhere in the trailer block** — between `Ext-ref:` and `Generated-by:`, or
+ between `Generated-by:` and `Change-Id:` — splits the footer, so the last paragraph is read as
+ body text.
- **Ending the message with a non-trailer line** — most often
`(cherry picked from commit <sha>)` on a backport. Put that line in its own paragraph *above* the
trailer block so the message still ends with a trailer.
If the trailer must be added after the fact, rebuild the whole message rather than appending a line:
-capture the `Change-Id` first, strip every `Change-Id:`/`Co-Authored-By:` line, then re-append both
-contiguously. Verify before pushing — this must print exactly `1`, matching the original:
+capture the `Change-Id` first, strip every `Ext-ref:`/`Generated-by:`/`Change-Id:` line, then
+re-append them contiguously. Verify before pushing — this must print exactly `1`, matching the
+original:
```bash
git log -1 --format=%B | grep -c '^Change-Id:'
(asterixdb) 05/06: [NO ISSUE][MISC] Advance commons-lang3 to 3.21.0
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit b96b4e824099a629a936f1a861a6547eacfff0c8
Author: Michael Blow <mi...@couchbase.com>
AuthorDate: Tue Sep 29 15:02:15 2026 -0400
[NO ISSUE][MISC] Advance commons-lang3 to 3.21.0
Ext-ref: MB-73268
Generated-by: Claude Opus 5.5 <no...@anthropic.com>
Change-Id: I0751745dfb7b51e6b3fb85fa0b149bd95b1a8d82
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21936
Tested-by: Michael Blow <mb...@apache.org>
Reviewed-by: Michael Blow <mb...@apache.org>
---
hyracks-fullstack/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hyracks-fullstack/pom.xml b/hyracks-fullstack/pom.xml
index 2c27a3df0c..eb935262aa 100644
--- a/hyracks-fullstack/pom.xml
+++ b/hyracks-fullstack/pom.xml
@@ -392,7 +392,7 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
- <version>3.20.0</version>
+ <version>3.21.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
(asterixdb) 02/06: [ASTERIXDB-3915][EXT]: Read wide Parquet decimals
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit beff29839981c825843c0377893680008c83247c
Author: Hussain Towaileb <Hu...@Couchbase.com>
AuthorDate: Mon Sep 28 10:20:54 2026 +0300
[ASTERIXDB-3915][EXT]: Read wide Parquet decimals
With decimal-to-double enabled, a FIXED_LEN_BYTE_ARRAY or BINARY decimal
of precision 19 or 20, or one stored wider than its precision needs, was
decoded through a long and lost its high bytes: a DECIMAL(20,5) of
123.456 read as 0.0. The long path is now chosen by the stored width
rather than the precision.
Added unit and runtime test cases covering FIXED_LEN_BYTE_ARRAY decimals
of several precisions and widths.
Ext-ref: MB-74262
Generated-by: Claude Opus 5.5 <no...@anthropic.com>
Change-Id: I796e444a73721ce39b16824dc8fc3555d1aa0b5e
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21915
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Hussain Towaileb <hu...@gmail.com>
Reviewed-by: Michael Blow <mb...@apache.org>
Reviewed-by: Ian Maxon <im...@apache.org>
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
---
.../external_dataset/ExternalDatasetTestUtils.java | 1 +
.../parquet/ParquetFileExampleGeneratorUtil.java | 46 +++++++
.../decimal-fixed-len-byte-array.1.ddl.sqlpp | 37 ++++++
.../decimal-fixed-len-byte-array.2.query.sqlpp | 26 ++++
.../decimal-fixed-len-byte-array.2.adm | 3 +
.../runtimets/testsuite_external_dataset_s3.xml | 8 ++
.../converter/primitve/DecimalConverter.java | 5 +-
.../hdfs/parquet/ParquetDecimalToDoubleTest.java | 140 +++++++++++++++++++++
8 files changed, 264 insertions(+), 2 deletions(-)
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/ExternalDatasetTestUtils.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/ExternalDatasetTestUtils.java
index 7963132494..cfe9492fe9 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/ExternalDatasetTestUtils.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/ExternalDatasetTestUtils.java
@@ -272,6 +272,7 @@ public class ExternalDatasetTestUtils {
loadData(generatedDataBasePath, "", "heterogeneous_1.parquet", definition, definitionSegment, false, false);
loadData(generatedDataBasePath, "", "heterogeneous_2.parquet", definition, definitionSegment, false, false);
loadData(generatedDataBasePath, "", "parquetTypes.parquet", definition, definitionSegment, false, false);
+ loadData(generatedDataBasePath, "", "parquetDecimals.parquet", definition, definitionSegment, false, false);
loadData(generatedDataBasePath, "", "friends.parquet", definition, definitionSegment, false, false);
}
diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/parquet/ParquetFileExampleGeneratorUtil.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/parquet/ParquetFileExampleGeneratorUtil.java
index 501fb277b0..71d5b01996 100644
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/parquet/ParquetFileExampleGeneratorUtil.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/parquet/ParquetFileExampleGeneratorUtil.java
@@ -27,6 +27,7 @@ import java.time.Duration;
import java.time.Instant;
import java.time.LocalDate;
import java.time.ZoneOffset;
+import java.util.Arrays;
import java.util.TimeZone;
import java.util.concurrent.TimeUnit;
@@ -68,6 +69,7 @@ public class ParquetFileExampleGeneratorUtil {
private static final int JULIAN_DAY_OF_EPOCH = 2440588;
private static final String FILE_NAME = "parquetTypes.parquet";
+ private static final String DECIMALS_FILE_NAME = "parquetDecimals.parquet";
private static final String SCHEMA = "message test { \n" + " required boolean boolean_field;\n"
+ " required int32 int8_field (INTEGER(8,true));\n"
@@ -99,6 +101,15 @@ public class ParquetFileExampleGeneratorUtil {
+ " required group mapField (MAP) {\n" + " repeated group key_value {\n"
+ " required int32 key;\n" + " required int32 value;\n" + " }\n" + " }" + "}";
+ // An unscaled value of precision 19 or 20 needs 9 bytes, and d10_wide is wider than its precision needs
+ private static final String DECIMALS_SCHEMA = "message decimals {\n" + " required int32 id;\n"
+ + " required fixed_len_byte_array(4) d9 (DECIMAL(9,2));\n"
+ + " required fixed_len_byte_array(8) d18 (DECIMAL(18,5));\n"
+ + " required fixed_len_byte_array(9) d19 (DECIMAL(19,4));\n"
+ + " required fixed_len_byte_array(9) d20 (DECIMAL(20,5));\n"
+ + " required fixed_len_byte_array(16) d38 (DECIMAL(38,10));\n"
+ + " required fixed_len_byte_array(12) d10_wide (DECIMAL(10,2));\n" + "}";
+
private ParquetFileExampleGeneratorUtil() {
}
@@ -142,6 +153,41 @@ public class ParquetFileExampleGeneratorUtil {
mapField.addGroup("key_value").append("key", 1).append("value", 1);
writer.write(message);
writer.close();
+ writeDecimalsExample(conf, root);
+ }
+
+ private static void writeDecimalsExample(Configuration conf, Path root) throws IOException {
+ MessageType schema = parseMessageType(DECIMALS_SCHEMA);
+ try (ParquetWriter<Group> writer =
+ ExampleParquetWriter.builder(new TestOutputFile(new Path(root, DECIMALS_FILE_NAME), conf))
+ .withType(schema).withCompressionCodec(UNCOMPRESSED).withConf(conf).build()) {
+ SimpleGroupFactory groupFactory = new SimpleGroupFactory(schema);
+ writer.write(decimalsRow(groupFactory, schema, 1, "1234567.89", "123.45600", "123.4560", "123.45600",
+ "123.4560000000", "99999999.99"));
+ writer.write(decimalsRow(groupFactory, schema, 2, "-1234567.89", "-9876543210.12345", "-123.4560",
+ "-123.45600", "-1.5000000000", "-99999999.99"));
+ writer.write(
+ decimalsRow(groupFactory, schema, 3, "9999999.99", "9999999999999.99999", "999999999999999.9999",
+ "-92233720368547.75809", "9999999999999999999999999999.9999999999", "0.01"));
+ }
+ }
+
+ private static Group decimalsRow(SimpleGroupFactory groupFactory, MessageType schema, int id, String... decimals) {
+ Group row = groupFactory.newGroup().append("id", id);
+ for (int i = 0; i < decimals.length; i++) {
+ int field = i + 1;
+ int width = schema.getType(field).asPrimitiveType().getTypeLength();
+ row.append(schema.getFieldName(field), toFixedLengthDecimal(new BigDecimal(decimals[i]), width));
+ }
+ return row;
+ }
+
+ private static Binary toFixedLengthDecimal(BigDecimal decimal, int width) {
+ byte[] minimal = decimal.unscaledValue().toByteArray();
+ byte[] bytes = new byte[width];
+ Arrays.fill(bytes, decimal.signum() < 0 ? (byte) 0xFF : 0);
+ System.arraycopy(minimal, 0, bytes, width - minimal.length, minimal.length);
+ return Binary.fromConstantByteArray(bytes);
}
private static int getSecondsSinceMidnight() {
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.1.ddl.sqlpp
new file mode 100644
index 0000000000..59e401beb6
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.1.ddl.sqlpp
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+DROP DATAVERSE test IF EXISTS;
+CREATE DATAVERSE test;
+
+USE test;
+
+
+CREATE TYPE ParquetType as {
+};
+
+CREATE EXTERNAL DATASET ParquetDataset(ParquetType) USING %adapter%
+(
+ %template%,
+ ("container"="playground"),
+ ("definition"="parquet-data/reviews"),
+ ("include"="*parquetDecimals.parquet"),
+ ("format" = "parquet"),
+ ("decimal-to-double" = "true")
+);
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.2.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.2.query.sqlpp
new file mode 100644
index 0000000000..4dda9b4a32
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.2.query.sqlpp
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+-- param max-warnings:json=1000
+
+USE test;
+
+SELECT p.id, p.d9, p.d18, p.d19, p.d20, p.d38, p.d10_wide
+FROM ParquetDataset p
+ORDER BY p.id;
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.2.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.2.adm
new file mode 100644
index 0000000000..08335cc705
--- /dev/null
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/common/parquet/parquet-types/decimal-fixed-len-byte-array/decimal-fixed-len-byte-array.2.adm
@@ -0,0 +1,3 @@
+{ "id": 1, "d9": 1234567.89, "d18": 123.456, "d19": 123.456, "d20": 123.456, "d38": 123.456, "d10_wide": 9.999999999E7 }
+{ "id": 2, "d9": -1234567.89, "d18": -9.87654321012345E9, "d19": -123.456, "d20": -123.456, "d38": -1.5, "d10_wide": -9.999999999E7 }
+{ "id": 3, "d9": 9999999.99, "d18": 1.0E13, "d19": 1.0E15, "d20": -9.223372036854777E13, "d38": 1.0E28, "d10_wide": 0.01 }
diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
index 859a7e55fb..cf9735a987 100644
--- a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
@@ -204,6 +204,14 @@
<expected-warn>Parquet decimal precision loss: precision '22' is greater than the maximum supported precision '20'</expected-warn>
</compilation-unit>
</test-case>
+ <test-case FilePath="external-dataset" check-warnings="true">
+ <compilation-unit name="common/parquet/parquet-types/decimal-fixed-len-byte-array">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/parquet-types/decimal-fixed-len-byte-array</output-dir>
+ <source-location>false</source-location>
+ <expected-warn>Parquet decimal precision loss: precision '38' is greater than the maximum supported precision '20'</expected-warn>
+ </compilation-unit>
+ </test-case>
<test-case FilePath="external-dataset" check-warnings="true">
<compilation-unit name="common/parquet/parquet-types/temporal">
<placeholder name="adapter" value="S3" />
diff --git a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/converter/primitve/DecimalConverter.java b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/converter/primitve/DecimalConverter.java
index e93bcf7c34..c7d2377b0c 100644
--- a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/converter/primitve/DecimalConverter.java
+++ b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/converter/primitve/DecimalConverter.java
@@ -55,10 +55,11 @@ public class DecimalConverter extends GenericPrimitiveConverter {
@Override
public void addBinary(Binary value) {
- if (precision <= LONG_MAX_PRECISION) {
+ // Decide by the stored width, not the precision: a precision of 19 or 20 needs 9 bytes, and a writer may
+ // use a FIXED_LEN_BYTE_ARRAY wider than the precision requires
+ if (value.length() <= Long.BYTES) {
addLong(getUnscaledLong(value.toByteBuffer()));
} else {
- //Unlimited precision
addConvertedDouble(new BigDecimal(new BigInteger(value.getBytes()), scale).doubleValue());
}
}
diff --git a/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/ParquetDecimalToDoubleTest.java b/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/ParquetDecimalToDoubleTest.java
new file mode 100644
index 0000000000..eae8e1b393
--- /dev/null
+++ b/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/ParquetDecimalToDoubleTest.java
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.asterix.external.input.record.reader.hdfs.parquet;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.File;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.asterix.dataflow.data.nontagged.serde.AObjectSerializerDeserializer;
+import org.apache.asterix.external.util.ExternalDataConstants.ParquetOptions;
+import org.apache.asterix.om.base.ADouble;
+import org.apache.asterix.om.base.ARecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hyracks.data.std.api.IValueReference;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroupFactory;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.util.HadoopOutputFile;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.schema.Types;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+/**
+ * Reads FIXED_LEN_BYTE_ARRAY-backed decimals with {@link ParquetOptions#DECIMAL_TO_DOUBLE} enabled.
+ */
+public class ParquetDecimalToDoubleTest {
+ private static final Column D9 = new Column("d9", 9, 2, 4, "1234567.89", "-1234567.89", "9999999.99", "-0.01");
+ private static final Column D18 =
+ new Column("d18", 18, 5, 8, "123.45600", "-9876543210.12345", "9999999999999.99999", "-0.00001");
+ // an unscaled value of precision 19 or 20 needs 9 bytes, and may not fit in a long
+ private static final Column D20 =
+ new Column("d20", 20, 5, 9, "123.45600", "-123.45600", "999999999999999.99999", "-92233720368547.75809");
+ private static final Column D38 = new Column("d38", 38, 10, 16, "123.4560000000", "-1.5000000000",
+ "9999999999999999999999999999.9999999999", "-0.0000000001");
+ // wider than precision 10 needs, as some writers produce
+ private static final Column D10_WIDE =
+ new Column("d10_wide", 10, 2, 12, "99999999.99", "-99999999.99", "0.01", "-21474836.48");
+ private static final Column[] COLUMNS = { D9, D18, D20, D38, D10_WIDE };
+ private static final int ROW_COUNT = 4;
+
+ @Rule
+ public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ @Test
+ public void testFixedLenByteArrayDecimals() throws Exception {
+ Configuration conf = new Configuration();
+ conf.set(ParquetOptions.HADOOP_DECIMAL_TO_DOUBLE, "true");
+ conf.set(ParquetOptions.HADOOP_TIMEZONE, "");
+ Path path = new Path(new File(tempFolder.getRoot(), "decimals.parquet").toURI());
+ write(path, conf);
+
+ int row = 0;
+ try (ParquetReader<IValueReference> reader =
+ ParquetReader.builder(new ParquetReadSupport(), path).withConf(conf).build()) {
+ for (IValueReference value = reader.read(); value != null; value = reader.read()) {
+ Map<String, Double> actual = toDoubles(value);
+ for (Column column : COLUMNS) {
+ String decimal = column.values[row];
+ Assert.assertEquals(column.name + " = " + decimal, new BigDecimal(decimal).doubleValue(),
+ actual.get(column.name), 0.0);
+ }
+ row++;
+ }
+ }
+ Assert.assertEquals(ROW_COUNT, row);
+ }
+
+ private static void write(Path path, Configuration conf) throws Exception {
+ Types.MessageTypeBuilder builder = Types.buildMessage();
+ for (Column column : COLUMNS) {
+ builder.required(PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY).length(column.width)
+ .as(LogicalTypeAnnotation.decimalType(column.scale, column.precision)).named(column.name);
+ }
+ MessageType schema = builder.named("decimals");
+ SimpleGroupFactory factory = new SimpleGroupFactory(schema);
+ try (ParquetWriter<Group> writer =
+ ExampleParquetWriter.builder(HadoopOutputFile.fromPath(path, conf)).withType(schema).build()) {
+ for (int row = 0; row < ROW_COUNT; row++) {
+ Group group = factory.newGroup();
+ for (Column column : COLUMNS) {
+ group.append(column.name, toFixedLength(new BigDecimal(column.values[row]), column.width));
+ }
+ writer.write(group);
+ }
+ }
+ }
+
+ private static Binary toFixedLength(BigDecimal decimal, int width) {
+ BigInteger unscaled = decimal.unscaledValue();
+ byte[] minimal = unscaled.toByteArray();
+ byte[] bytes = new byte[width];
+ Arrays.fill(bytes, unscaled.signum() < 0 ? (byte) 0xFF : 0);
+ System.arraycopy(minimal, 0, bytes, width - minimal.length, minimal.length);
+ return Binary.fromConstantByteArray(bytes);
+ }
+
+ private static Map<String, Double> toDoubles(IValueReference value) throws Exception {
+ DataInputStream in = new DataInputStream(
+ new ByteArrayInputStream(value.getByteArray(), value.getStartOffset(), value.getLength()));
+ ARecord record = (ARecord) AObjectSerializerDeserializer.INSTANCE.deserialize(in);
+ Map<String, Double> doubles = new HashMap<>();
+ for (int i = 0; i < record.numberOfFields(); i++) {
+ doubles.put(record.getType().getFieldNames()[i], ((ADouble) record.getValueByPos(i)).getDoubleValue());
+ }
+ return doubles;
+ }
+
+ private record Column(String name, int precision, int scale, int width, String... values) {
+ }
+}
(asterixdb) 06/06: Merge branch 'gerrit/morpheus' into 'master'
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit d126525a77a98a3631f6b9f5e61844e6c676d5e3
Merge: 84e7e4ec30 b96b4e8240
Author: Michael Blow <mi...@couchbase.com>
AuthorDate: Tue Sep 29 18:59:38 2026 -0400
Merge branch 'gerrit/morpheus' into 'master'
* [NO ISSUE][MISC] Record AI work with Generated-by (MB-74304)
* [ASTERIXDB-3915][EXT]: Read wide Parquet decimals (MB-74262)
* [NO ISSUE][MISC] Advance the AWS SDK to 2.55.7 and woodstox to 7.3.0
(MB-73268)
* [NO ISSUE][MISC] Advance commons-lang3 to 3.21.0 (MB-73268)
The new decimal test case sits beside this branch's own Parquet decimal
case, and hyracks-fullstack takes the same AWS SDK version per its
keep-in-sync note.
Ext-ref: MB-74304,MB-74262,MB-73268
Generated-by: Claude Opus 5.5 <no...@anthropic.com>
Change-Id: I289dc1899c490a779ce18090e401c1967e487ae4
.../external_dataset/ExternalDatasetTestUtils.java | 1 +
.../parquet/ParquetFileExampleGeneratorUtil.java | 46 +++++++
.../decimal-fixed-len-byte-array.1.ddl.sqlpp | 37 ++++++
.../decimal-fixed-len-byte-array.2.query.sqlpp | 26 ++++
.../decimal-fixed-len-byte-array.2.adm | 3 +
.../runtimets/testsuite_external_dataset_s3.xml | 8 ++
.../converter/primitve/DecimalConverter.java | 5 +-
.../hdfs/parquet/ParquetDecimalToDoubleTest.java | 140 +++++++++++++++++++++
asterixdb/pom.xml | 6 +-
hyracks-fullstack/pom.xml | 4 +-
10 files changed, 269 insertions(+), 7 deletions(-)
diff --cc asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/ExternalDatasetTestUtils.java
index 2727e2d559,cfe9492fe9..2f0c3f7ea8
--- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/ExternalDatasetTestUtils.java
+++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/ExternalDatasetTestUtils.java
@@@ -520,192 -272,8 +520,193 @@@ public class ExternalDatasetTestUtils
loadData(generatedDataBasePath, "", "heterogeneous_1.parquet", definition, definitionSegment, false, false);
loadData(generatedDataBasePath, "", "heterogeneous_2.parquet", definition, definitionSegment, false, false);
loadData(generatedDataBasePath, "", "parquetTypes.parquet", definition, definitionSegment, false, false);
+ loadData(generatedDataBasePath, "", "parquetDecimals.parquet", definition, definitionSegment, false, false);
loadData(generatedDataBasePath, "", "friends.parquet", definition, definitionSegment, false, false);
+
+ loadData(PARQUET_BASEDIR, "", "array_of_primitives.parquet", PARQUET_NULL_TEST_DIRECTORY, definitionSegment,
+ false, false);
+ loadData(PARQUET_BASEDIR, "", "array_of_single_field_struct.parquet", PARQUET_NULL_TEST_DIRECTORY,
+ definitionSegment, false, false);
+ loadData(PARQUET_BASEDIR, "", "array_of_structs_with_nulls.parquet", PARQUET_NULL_TEST_DIRECTORY,
+ definitionSegment, false, false);
+ loadData(PARQUET_BASEDIR, "", "repeated_field.parquet", PARQUET_NULL_TEST_DIRECTORY, definitionSegment, false,
+ false);
+ loadData(PARQUET_BASEDIR, "", "repeated_struct.parquet", PARQUET_NULL_TEST_DIRECTORY, definitionSegment, false,
+ false);
+
+ loadRowGroupsFile();
+ loadTemporalTypesFile();
+
+ Collection<File> files =
+ IoUtil.getMatchingFiles(Paths.get(generatedDataBasePath + "/external-filter"), PARQUET_FILTER);
+ for (File file : files) {
+ String fileName = file.getName();
+ String externalFilterDefinition = file.getParent().substring(generatedDataBasePath.length() + 1) + "/";
+ loadData(file.getParent(), "", fileName, "parquet-data/" + externalFilterDefinition, "", false, false);
+ }
+ }
+
+ private static void loadAvroFiles() {
+ String generatedDataBasePath = AVRO_GEN_BASEDIR;
+ String definition = AVRO_DEFINITION;
+ String definitionSegment = "";
+
+ loadData(generatedDataBasePath, "", "dummy_tweet.avro", definition, definitionSegment, false, false);
+ loadData(generatedDataBasePath, "", "id_age.avro", definition, definitionSegment, false, false);
+ loadData(generatedDataBasePath, "", "id_age-string.avro", definition, definitionSegment, false, false);
+ loadData(generatedDataBasePath, "", "id_name.avro", definition, definitionSegment, false, false);
+ loadData(generatedDataBasePath, "", "id_name_comment.avro", definition, definitionSegment, false, false);
+ loadData(generatedDataBasePath, "", "heterogeneous_1.avro", definition, definitionSegment, false, false);
+ loadData(generatedDataBasePath, "", "heterogeneous_2.avro", definition, definitionSegment, false, false);
+ loadData(generatedDataBasePath, "", "avro_type.avro", definition, definitionSegment, false, false);
+ loadData(generatedDataBasePath, "", "partition_heterogeneous.avro", definition, definitionSegment, false,
+ false);
+ loadData(generatedDataBasePath, "", "avro_logical_type.avro", definition, definitionSegment, false, false);
+
+ Collection<File> files =
+ IoUtil.getMatchingFiles(Paths.get(generatedDataBasePath + "/external-filter"), AVRO_FILTER);
+ for (File file : files) {
+ String fileName = file.getName();
+ String externalFilterDefinition = file.getParent().substring(generatedDataBasePath.length() + 1) + "/";
+ loadData(file.getParent(), "", fileName, "avro-data/" + externalFilterDefinition, "", false, false);
+ }
+ }
+
+ private static void loadDeltaTableFiles() {
+ String generatedDataBasePath = DELTA_GEN_BASEDIR;
+ loadDeltaDirectory(generatedDataBasePath, "/empty_delta_table", PARQUET_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/empty_delta_table/_delta_log", JSON_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/modified_delta_table", PARQUET_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/modified_delta_table/_delta_log", JSON_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/multiple_file_delta_table", PARQUET_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/multiple_file_delta_table/_delta_log", JSON_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/delta_all_type/_delta_log", JSON_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/delta_all_type", PARQUET_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/delta_file_size_nine/_delta_log", JSON_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/delta_file_size_nine", PARQUET_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/delta_file_size_one/_delta_log", JSON_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/delta_file_size_one", PARQUET_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/partitioned_delta_table", PARQUET_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/partitioned_delta_table/_delta_log", JSON_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/timestamp_partitioned_delta_table", PARQUET_FILTER, "delta-data/");
+ loadDeltaDirectory(generatedDataBasePath, "/timestamp_partitioned_delta_table/_delta_log", JSON_FILTER,
+ "delta-data/");
+ }
+
+ private static void loadDeltaDirectory(String dataBasePath, String rootPath, FilenameFilter filter,
+ String definitionPart) {
+ Collection<File> files = IoUtil.getMatchingFiles(Paths.get(dataBasePath + rootPath), filter);
+ for (File file : files) {
+ String fileName = file.getName();
+ String externalFilterDefinition = file.getParent().substring(dataBasePath.length() + 1) + "/";
+ loadData(file.getParent(), "", fileName, definitionPart + externalFilterDefinition, "", false, false);
+ }
+ }
+
+ private static void loadDirectory(String dataBasePath, String rootPath, FilenameFilter filter) {
+ File dir = new File(dataBasePath, rootPath);
+ if (!dir.exists() || !dir.isDirectory()) {
+ return;
+ }
+
+ Collection<File> files = IoUtil.getMatchingFiles(dir.toPath(), filter);
+ int size = 0;
+ for (File file : files) {
+ String path = file.getPath();
+ // +1 to remove the leading '/'
+ int startIndex = path.indexOf(rootPath) + rootPath.length() + 1;
+ int endIndex = path.lastIndexOf(File.separatorChar);
+ String definitionSegment = rootPath + File.separator + path.substring(startIndex, endIndex);
+ loadData(path.substring(0, endIndex), "", file.getName(), "", definitionSegment, false, false);
+ size++;
+ }
+ LOGGER.info("Loaded {} files from {}", size, dataBasePath + File.separator + rootPath);
+ }
+
+ /**
+ * Writes and uploads one Parquet file carrying a date, a time and a UTC-adjusted microsecond timestamp, so the
+ * date-to-int, time-to-int and timestamp-to-long options can be observed against the same rows. Every {@code ts}
+ * carries a sub-millisecond remainder, which is exactly what the reader discards when the value is read as a
+ * datetime and keeps when it is read as its stored number.
+ */
+ private static void loadTemporalTypesFile() {
+ try {
+ MessageType schema = Types.buildMessage().required(PrimitiveType.PrimitiveTypeName.INT64).named("id")
+ .required(PrimitiveType.PrimitiveTypeName.INT32).as(LogicalTypeAnnotation.dateType()).named("d")
+ .required(PrimitiveType.PrimitiveTypeName.INT32)
+ .as(LogicalTypeAnnotation.timeType(false, LogicalTypeAnnotation.TimeUnit.MILLIS)).named("t")
+ .required(PrimitiveType.PrimitiveTypeName.INT64)
+ .as(LogicalTypeAnnotation.timestampType(true, LogicalTypeAnnotation.TimeUnit.MICROS)).named("ts")
+ .named("row");
+ File directory = new File(BINARY_GEN_BASEDIR);
+ java.nio.file.Files.createDirectories(directory.toPath());
+ File file = new File(directory, "temporal_types.parquet");
+ java.nio.file.Files.deleteIfExists(file.toPath());
+ Configuration configuration = new Configuration();
+ GroupWriteSupport.setSchema(schema, configuration);
+ SimpleGroupFactory groups = new SimpleGroupFactory(schema);
+ org.apache.hadoop.fs.Path path = new org.apache.hadoop.fs.Path(file.getAbsolutePath());
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(path).withType(schema)
+ .withConf(configuration).withCompressionCodec(CompressionCodecName.UNCOMPRESSED).build()) {
+ for (int i = 1; i <= TEMPORAL_TYPES_ROW_COUNT; i++) {
+ writer.write(groups.newGroup().append("id", (long) i).append("d", TEMPORAL_TYPES_EPOCH_DAY + i)
+ .append("t", TEMPORAL_TYPES_MILLIS_OF_DAY + i)
+ .append("ts", TimeUnit.MILLISECONDS.toMicros(i) + ROW_GROUPS_SUBMILLI_MICROS));
+ }
+ }
+ loadData(BINARY_GEN_BASEDIR, "", "temporal_types.parquet", PARQUET_TEMPORAL_TYPES_DIRECTORY, "", false,
+ false);
+ } catch (IOException e) {
+ throw new IllegalStateException("failed to prepare the parquet temporal-types fixture", e);
+ }
+ }
+
+ /**
+ * Writes and uploads a single Parquet file of {@value #ROW_GROUPS_ROW_COUNT} rows with a deliberately small
+ * row-group size, so it holds many row groups rather than the one every other Parquet fixture here has. A file
+ * with one row group cannot show row-group skipping: either that group is read or the file is, and the two are
+ * the same thing.
+ * <p>
+ * {@code id} ascends, so each row group's statistics cover a distinct range and a predicate is decidable from
+ * them alone. {@code ts} is microseconds and every row carries a sub-millisecond remainder, which the reader
+ * truncates away; a predicate written against what the collection displays must therefore still match it.
+ */
+ private static void loadRowGroupsFile() {
+ try {
+ MessageType schema = Types.buildMessage().required(PrimitiveType.PrimitiveTypeName.INT64).named("id")
+ .required(PrimitiveType.PrimitiveTypeName.INT64)
+ .as(LogicalTypeAnnotation.timestampType(true, LogicalTypeAnnotation.TimeUnit.MICROS)).named("ts")
+ .named("row");
+ File directory = new File(BINARY_GEN_BASEDIR);
+ java.nio.file.Files.createDirectories(directory.toPath());
+ File file = new File(directory, "row_groups.parquet");
+ java.nio.file.Files.deleteIfExists(file.toPath());
+ Configuration configuration = new Configuration();
+ GroupWriteSupport.setSchema(schema, configuration);
+ SimpleGroupFactory groups = new SimpleGroupFactory(schema);
+ org.apache.hadoop.fs.Path path = new org.apache.hadoop.fs.Path(file.getAbsolutePath());
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(path).withType(schema)
+ .withConf(configuration).withRowGroupSize((long) ROW_GROUPS_ROW_GROUP_BYTES)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED).withDictionaryEncoding(false).build()) {
+ for (int i = 1; i <= ROW_GROUPS_ROW_COUNT; i++) {
+ long micros = TimeUnit.MILLISECONDS.toMicros(i) + ROW_GROUPS_SUBMILLI_MICROS;
+ writer.write(groups.newGroup().append("id", (long) i).append("ts", micros));
+ }
+ }
+ int rowGroups;
+ try (ParquetFileReader reader = ParquetFileReader.open(configuration, path)) {
+ rowGroups = reader.getFooter().getBlocks().size();
+ }
+ // Without this the suite would pass while testing nothing it was written for: one row group makes every
+ // query below read the whole file, which is what the failing case looks like.
+ if (rowGroups < 2) {
+ throw new IllegalStateException("row-groups fixture collapsed to " + rowGroups + " row group(s)");
+ }
+ LOGGER.info("Parquet row-groups fixture: {} rows in {} row groups", ROW_GROUPS_ROW_COUNT, rowGroups);
+ loadData(BINARY_GEN_BASEDIR, "", "row_groups.parquet", PARQUET_ROW_GROUPS_DIRECTORY, "", false, false);
+ } catch (IOException e) {
+ throw new IllegalStateException("failed to prepare the parquet row-groups fixture", e);
+ }
}
private static void loadData(String fileBasePath, String filePathSegment, String filename, String definition,
diff --cc asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
index b06f4f272a,cf9735a987..4c6f20995e
--- a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
@@@ -571,168 -77,10 +571,176 @@@
</compilation-unit>
</test-case>
<!-- Parquet Tests Start -->
+ <test-group name="parquet">
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/select-all-fields">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/select-all-fields</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/select-count-one-field">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/select-count-one-field</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/multi-file-multi-schema">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/multi-file-multi-schema</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/field-access-pushdown">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/field-access-pushdown</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/ASTERIXDB-3540">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Clean-JSON">common/parquet/ASTERIXDB-3540</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/null-in-array">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Clean-JSON">common/parquet/null-in-array</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/array-access-pushdown">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/array-access-pushdown</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/missing-fields">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/missing-fields</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/string-standard-utf8">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/string-standard-utf8</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/object-concat">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/object-concat</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset" check-warnings="true">
+ <compilation-unit name="common/parquet/type-mismatch">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/type-mismatch</output-dir>
+ <expected-warn>ASX0002: Type mismatch: function field-access-by-name expects its 1st input parameter to be of type object, but the actual input type is array (in line 31, at column 29)</expected-warn>
+ <expected-warn>ASX0002: Type mismatch: function array-star expects its 1st input parameter to be of type array, but the actual input type is object (in line 31, at column 24)</expected-warn>
+ <expected-warn>ASX0002: Type mismatch: function field-access-by-name expects its 1st input parameter to be of type object, but the actual input type is array (in line 31, at column 52)</expected-warn>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/heterogeneous-access-pushdown">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/heterogeneous-access-pushdown</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/pushdown-plans">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/pushdown-plans</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset/s3">
+ <compilation-unit name="parquet-anonymous-access">
+ <output-dir compare="Text">parquet-anonymous-access</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset/s3">
+ <compilation-unit name="parquet-temporary-access">
+ <output-dir compare="Text">parquet-temporary-access</output-dir>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/invalid-type">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">none</output-dir>
+ <expected-error>ASX1161: Type 'ParquetType' contains declared fields, which is not supported for 'parquet' format</expected-error>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset" check-warnings="true">
+ <compilation-unit name="common/parquet/no-files">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/no-files</output-dir>
+ <source-location>false</source-location>
+ <expected-warn>The provided external dataset configuration returned no files from the external source</expected-warn>
+ <expected-warn>The provided external dataset configuration returned no files from the external source</expected-warn>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/invalid-parquet-files">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">none</output-dir>
+ <source-location>false</source-location>
+ <expected-error>20-records.json. Reason: not a Parquet file</expected-error>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset" check-warnings="true">
+ <compilation-unit name="common/parquet/parquet-types/unset-flags">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/parquet-types/unset-flags</output-dir>
+ <source-location>false</source-location>
+ <expected-warn>Parquet file(s) contain unsigned integer that is larger than the 'bigint' range</expected-warn>
+ <expected-warn>Parquet file(s) contain values of the temporal type 'datetime' that are adjusted to UTC. Recreate the external dataset and set the option 'timezone' to get the local-adjusted 'datetime' value</expected-warn>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset" check-warnings="true">
+ <compilation-unit name="common/parquet/parquet-types/decimal">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/parquet-types/decimal</output-dir>
+ <source-location>false</source-location>
+ <expected-error>ASX0054: Parquet type 'required int32 decimal32_field (DECIMAL(5,4))' is not supported by default. To enable type conversion, recreate the external dataset with the option 'decimal-to-double' enabled</expected-error>
+ <expected-warn>Parquet decimal precision loss: precision '22' is greater than the maximum supported precision '20'</expected-warn>
+ </compilation-unit>
+ </test-case>
++ <test-case FilePath="external-dataset" check-warnings="true">
++ <compilation-unit name="common/parquet/parquet-types/decimal-fixed-len-byte-array">
++ <placeholder name="adapter" value="S3" />
++ <output-dir compare="Text">common/parquet/parquet-types/decimal-fixed-len-byte-array</output-dir>
++ <source-location>false</source-location>
++ <expected-warn>Parquet decimal precision loss: precision '38' is greater than the maximum supported precision '20'</expected-warn>
++ </compilation-unit>
++ </test-case>
+ <test-case FilePath="external-dataset" check-warnings="true">
+ <compilation-unit name="common/parquet/parquet-types/temporal">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/parquet-types/temporal</output-dir>
+ <source-location>false</source-location>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset">
+ <compilation-unit name="common/parquet/parquet-types/invalid-timezone">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/parquet-types/invalid-timezone</output-dir>
+ <source-location>false</source-location>
+ <expected-error>Provided timezone is invalid: 'invalid-timezone'</expected-error>
+ </compilation-unit>
+ </test-case>
+ <test-case FilePath="external-dataset" check-warnings="true">
+ <compilation-unit name="common/parquet/parquet-types/disable-json-parsing">
+ <placeholder name="adapter" value="S3" />
+ <output-dir compare="Text">common/parquet/parquet-types/disable-json-parsing</output-dir>
+ <source-location>false</source-location>
+ </compilation-unit>
+ </test-case>
+ </test-group>
+ <!-- Deltalake Tests Start -->
<test-case FilePath="external-dataset">
- <compilation-unit name="common/parquet/select-all-fields">
+ <compilation-unit name="common/deltalake-multiple-file-read">
<placeholder name="adapter" value="S3" />
- <output-dir compare="Text">common/parquet/select-all-fields</output-dir>
+ <output-dir compare="Text">common/deltalake-multiple-file-read</output-dir>
</compilation-unit>
</test-case>
<test-case FilePath="external-dataset">
diff --cc asterixdb/pom.xml
index fd3b50a192,4216a1ff33..78af3dc704
--- a/asterixdb/pom.xml
+++ b/asterixdb/pom.xml
@@@ -104,11 -98,6 +104,11 @@@
<parquet.version>1.18.1</parquet.version> <!-- NOTICE: please update transitives from parquet below on any change -->
<hadoop-awsjavasdk.version>1.12.797</hadoop-awsjavasdk.version>
+ <!-- keep in sync with hyracks-fullstack/pom.xml -->
- <awsjavasdk.version>2.55.5</awsjavasdk.version>
++ <awsjavasdk.version>2.55.7</awsjavasdk.version>
+ <!-- @derived-from: software.amazon.awssdk:aws-sdk-java-pom:${awsjavasdk.version}#awscrt.version -->
+ <awsjavasdk.crt.version>0.48.4</awsjavasdk.crt.version>
+
<azureblobjavasdk.version>12.35.1</azureblobjavasdk.version>
<azurecommonjavasdk.version>12.34.1</azurecommonjavasdk.version>
<azureidentity.version>1.18.6</azureidentity.version>
diff --cc hyracks-fullstack/pom.xml
index 32e5e8b242,eb935262aa..1fdcffe5de
--- a/hyracks-fullstack/pom.xml
+++ b/hyracks-fullstack/pom.xml
@@@ -76,20 -81,7 +76,20 @@@
<jackson.version>2.22.3</jackson.version>
<jackson-annotations.version>2.22</jackson-annotations.version>
<jackson-databind.version>${jackson.version}</jackson-databind.version>
+ <!-- 4.2 is a new netty generation; stay on the 4.1.x line (patch updates only) -->
<netty.version>4.1.138.Final</netty.version>
+ <netty.tcnative.version>2.0.84.Final</netty.tcnative.version>
+ <asm.version>9.10.1</asm.version>
+ <!-- keep in sync with asterixdb/pom.xml -->
- <awsjavasdk.version>2.55.5</awsjavasdk.version>
++ <awsjavasdk.version>2.55.7</awsjavasdk.version>
+
+ <!-- keep in sync with asterixdb/pom.xml -->
+ <google-cloud-storage.version>2.74.0</google-cloud-storage.version>
+ <!-- @derived-from: com.google.cloud:google-cloud-storage:${google-cloud-storage.version}#dep:google-cloud-core -->
+ <google-cloud-core.version>2.76.0</google-cloud-core.version>
+
+ <!-- keep in sync with asterixdb/pom.xml -->
+ <azurecore.version>1.59.1</azurecore.version>
<implementation.title>Apache Hyracks and Algebricks - ${project.name}</implementation.title>
<implementation.url>https://asterixdb.apache.org/</implementation.url>
(asterixdb) 03/06: [NO ISSUE][MISC] Advance the AWS SDK to 2.55.7 and woodstox to 7.3.0
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit be88b4bb261f0bd50e352da27c939ec52250b856
Author: Michael Blow <mi...@couchbase.com>
AuthorDate: Tue Sep 29 12:31:57 2026 -0400
[NO ISSUE][MISC] Advance the AWS SDK to 2.55.7 and woodstox to 7.3.0
Ext-ref: MB-73268
Generated-by: Claude Opus 5.5 <no...@anthropic.com>
Change-Id: I9237a9137923e3b187e0a66c25e292afd8e44ac6
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21912
Tested-by: Michael Blow <mb...@apache.org>
Reviewed-by: Michael Blow <mb...@apache.org>
---
asterixdb/pom.xml | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/asterixdb/pom.xml b/asterixdb/pom.xml
index 54f1741825..4216a1ff33 100644
--- a/asterixdb/pom.xml
+++ b/asterixdb/pom.xml
@@ -93,7 +93,7 @@
<hadoop.version>3.4.3</hadoop.version>
<jacoco.version>0.7.6.201602180812</jacoco.version>
<log4j.version>2.26.1</log4j.version>
- <awsjavasdk.version>2.55.5</awsjavasdk.version>
+ <awsjavasdk.version>2.55.7</awsjavasdk.version>
<s3mock.version>5.2.3</s3mock.version>
<parquet.version>1.18.1</parquet.version> <!-- NOTICE: please update transitives from parquet below on any change -->
<hadoop-awsjavasdk.version>1.12.797</hadoop-awsjavasdk.version>
@@ -1224,12 +1224,12 @@
<dependency>
<groupId>org.codehaus.woodstox</groupId>
<artifactId>stax2-api</artifactId>
- <version>4.3.0</version>
+ <version>4.3.1</version>
</dependency>
<dependency>
<groupId>com.fasterxml.woodstox</groupId>
<artifactId>woodstox-core</artifactId>
- <version>7.2.2</version>
+ <version>7.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
(asterixdb) 04/06: Merge branch 'gerrit/trinity' into 'gerrit/morpheus'
Posted by mb...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/asterixdb.git
commit a72cb245232d8dce19133f8b1cbda4896d0a330d
Merge: d336b0f967 be88b4bb26
Author: Michael Blow <mi...@couchbase.com>
AuthorDate: Tue Sep 29 14:18:29 2026 -0400
Merge branch 'gerrit/trinity' into 'gerrit/morpheus'
* [NO ISSUE][MISC] Record AI work with Generated-by (MB-74304)
* [ASTERIXDB-3915][EXT]: Read wide Parquet decimals (MB-74262)
* [NO ISSUE][MISC] Advance the AWS SDK to 2.55.7 and woodstox to 7.3.0
(MB-73268)
Ext-ref: MB-74304,MB-74262,MB-73268
Generated-by: Claude Opus 5.5 <no...@anthropic.com>
Change-Id: I54a18a1a4d812f36002d41594679c5267554c6b7
.../external_dataset/ExternalDatasetTestUtils.java | 1 +
.../parquet/ParquetFileExampleGeneratorUtil.java | 46 +++++++
.../decimal-fixed-len-byte-array.1.ddl.sqlpp | 37 ++++++
.../decimal-fixed-len-byte-array.2.query.sqlpp | 26 ++++
.../decimal-fixed-len-byte-array.2.adm | 3 +
.../runtimets/testsuite_external_dataset_s3.xml | 8 ++
.../converter/primitve/DecimalConverter.java | 5 +-
.../hdfs/parquet/ParquetDecimalToDoubleTest.java | 140 +++++++++++++++++++++
asterixdb/pom.xml | 6 +-
9 files changed, 267 insertions(+), 5 deletions(-)