You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xi...@apache.org on 2026/07/19 22:03:56 UTC
(pinot) branch master updated: Upgrade Pinot to JDK 25 (#19014)
This is an automated email from the ASF dual-hosted git repository.
xiangfu0 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 28922369c88 Upgrade Pinot to JDK 25 (#19014)
28922369c88 is described below
commit 28922369c889b448aaef396681b2966d1fb2a138
Author: Xiaotian (Jackie) Jiang <17...@users.noreply.github.com>
AuthorDate: Sun Jul 19 15:03:49 2026 -0700
Upgrade Pinot to JDK 25 (#19014)
* Bump org.apache.datasketches:datasketches-java from 6.2.0 to 9.0.0
* Require JDK 25 baseline and port to datasketches-java 9.0.0
datasketches-java 9.0.0 ships Java 25 bytecode (its foreign-memory access
layer needs the FFM API finalized after Java 21), so the two changes are
inseparable: Pinot services now build and run on JDK 25+.
- pom: jdk.version 21 -> 25; enforcer rule and message updated; SPI/client
modules keep their hard-coded Java 11 bytecode targets
- Port all datasketches call sites (pinot-segment-local, pinot-core,
pinot-integration-tests) to the 9.0.0 API: theta/tuple/frequencies class
renames, tuple.aninteger.IntegerSketch -> IntegerTupleSketch, removed
Sketches utility classes, datasketches-memory Memory ->
java.lang.foreign.MemorySegment
- CI: unit/integration/quickstart/compat workflows to temurin-25; docker
publish matrices drop the no-longer-buildable JDK 21 variant; Dockerfile,
Dockerfile.build, Dockerfile.package defaults to 25
- compatibility-verifier: stop forcing -Djdk.version=21 on every build;
each tree now builds with its own pom default (release-1.5.0 targets 11,
current targets 25) - verified locally that release-1.5.0 builds on JDK 25
- Add DataSketchesSerialCompatTest: pinned Base64 fixtures generated by
datasketches-java 6.2.0 for theta (ordered/unordered), integer tuple,
KLL doubles, CPC, frequent longs/strings, asserted through the query-path
ObjectSerDe instances including non-zero-position ByteBuffer wire paths
- LICENSE-binary: datasketches-java 9.0.0; datasketches-memory removed
(9.0.0 no longer depends on it); docs (README, helm, CLAUDE.md) updated
* Replace removed System.setSecurityManager usage in predownload test
JEP 486 (JDK 24+) made System.setSecurityManager throw
UnsupportedOperationException unconditionally. The test already intercepts
System.exit via ExitHelper.setExitAction; restore the default action in the
finally block instead of the vestigial SecurityManager save/restore.
* Add BenchmarkSketchSerDe covering sketch serde deserialize+merge hot paths
Benchmarks the per-row ObjectSerDeUtils sketch deserialize + union/merge
paths (theta, tuple, KLL, CPC) used by aggregation over serialized-sketch
BYTES columns, to compare datasketches memory access layers across
upgrades (6.2.0 Unsafe vs 9.0.0 java.lang.foreign MemorySegment).
* Fix Comparable type mismatch in SerializedFrequent*Sketch wrappers
The wrappers declared Comparable<sketch type> while runtime comparisons
(final-result ordering) pass wrapper instances, so the erased compareTo
bridge would throw ClassCastException. Align with every other Serialized*
custom object: implement Comparable<wrapper> and use Integer.compare.
Pre-existing issue surfaced by review on the datasketches type rename.
* Address review comments: parameterize remaining raw tuple types, fix stale JDK 21 doc mentions
---------
Co-authored-by: Xiang Fu <xi...@gmail.com>
---
.github/workflows/build-all-pinot-docker-image.yml | 6 +-
.github/workflows/build-pinot-docker-image.yml | 2 +-
.../nightly-build-all-pinot-docker-image.yml | 6 +-
.github/workflows/pinot_compatibility_checks.yml | 12 +-
.github/workflows/pinot_compatibility_tests.yml | 4 +-
.github/workflows/pinot_integration_tests.yml | 2 +-
...ulti_stage_query_engine_compatibility_tests.yml | 4 +-
.github/workflows/pinot_quickstart_tests.yml | 2 +-
.github/workflows/pinot_unit_tests.yml | 6 +-
.../docker/.pinot_docker_image_build_and_push.sh | 2 +-
...not_multi_arch_docker_image_manifest_package.sh | 2 +-
.../docker/.pinot_package_single_platform.sh | 6 +-
CLAUDE.md | 4 +-
LICENSE-binary | 3 +-
README.md | 2 +-
compatibility-verifier/checkoutAndBuild.sh | 5 +-
docker/images/pinot-base/README.md | 10 +-
docker/images/pinot/Dockerfile | 4 +-
docker/images/pinot/Dockerfile.build | 4 +-
docker/images/pinot/Dockerfile.package | 2 +-
helm/pinot/values.yaml | 8 +-
.../apache/pinot/core/common/ObjectSerDeUtils.java | 87 ++--
.../core/function/scalar/SketchFunctions.java | 85 ++--
...ValueIntegerTupleSketchAggregationFunction.java | 4 +-
.../DistinctCountCPCSketchAggregationFunction.java | 4 +-
...inctCountRawThetaSketchAggregationFunction.java | 6 +-
...istinctCountThetaSketchAggregationFunction.java | 184 +++----
.../FrequentLongsSketchAggregationFunction.java | 59 +--
.../FrequentStringsSketchAggregationFunction.java | 57 +--
.../IntegerTupleSketchAggregationFunction.java | 22 +-
.../function/PercentileKLLAggregationFunction.java | 4 +-
...aluesIntegerTupleSketchAggregationFunction.java | 4 +-
.../FunnelCountAggregationFunctionFactory.java | 14 +-
.../funnel/ThetaSketchAggregationStrategy.java | 25 +-
.../function/funnel/ThetaSketchMergeStrategy.java | 22 +-
.../ThetaSketchResultExtractionStrategy.java | 13 +-
.../DistinctCountThetaSketchAggregator.java | 16 +-
.../aggregator/IntegerTupleSketchAggregator.java | 16 +-
.../core/common/DataSketchesSerialCompatTest.java | 540 +++++++++++++++++++++
.../pinot/core/common/ObjectSerDeUtilsTest.java | 26 +-
.../core/function/scalar/SketchFunctionsTest.java | 9 +-
...tinctCountCPCSketchAggregationFunctionTest.java | 4 +-
.../DistinctCountThetaSketchAggregatorTest.java | 26 +-
.../IntegerTupleSketchAggregatorTest.java | 22 +-
...ctCountIntegerSumTupleSketchStarTreeV2Test.java | 18 +-
.../v2/DistinctCountThetaSketchStarTreeV2Test.java | 14 +-
.../DistinctCountThetaSketchQueriesTest.java | 15 +-
.../queries/FrequentItemsSketchQueriesTest.java | 46 +-
.../queries/FunnelCountQueriesThetaSketchTest.java | 4 +-
.../integration/tests/custom/ThetaSketchTest.java | 8 +-
.../integration/tests/custom/TupleSketchTest.java | 20 +-
.../apache/pinot/perf/BenchmarkSketchSerDe.java | 151 ++++++
.../DistinctCountThetaSketchValueAggregator.java | 56 +--
.../IntegerTupleSketchValueAggregator.java | 44 +-
.../SerializedFrequentLongsSketch.java | 12 +-
.../SerializedFrequentStringsSketch.java | 12 +-
.../local/customobject/ThetaSketchAccumulator.java | 26 +-
.../customobject/TupleIntSketchAccumulator.java | 20 +-
.../segment/local/utils/CustomSerDeUtils.java | 35 +-
...istinctCountThetaSketchValueAggregatorTest.java | 89 ++--
.../IntegerTupleSketchValueAggregatorTest.java | 30 +-
.../customobject/ThetaSketchAccumulatorTest.java | 31 +-
.../TupleIntSketchAccumulatorTest.java | 24 +-
.../PredownloadPredownloadStatusRecorderTest.java | 4 +-
pom.xml | 10 +-
65 files changed, 1357 insertions(+), 657 deletions(-)
diff --git a/.github/workflows/build-all-pinot-docker-image.yml b/.github/workflows/build-all-pinot-docker-image.yml
index 22433cfbfa1..b645d679bc0 100644
--- a/.github/workflows/build-all-pinot-docker-image.yml
+++ b/.github/workflows/build-all-pinot-docker-image.yml
@@ -53,7 +53,7 @@ jobs:
strategy:
fail-fast: false
matrix:
- java-version: ["21", "25"]
+ java-version: ["25"]
name: Compile JDK-${{ matrix.java-version }} Pinot
needs: [generate-build-info]
steps:
@@ -77,7 +77,7 @@ jobs:
fail-fast: false
matrix:
java-disto: ["ms-openjdk", "amazoncorretto"]
- java-version: ["21", "25"]
+ java-version: ["25"]
arch: ["amd64", "arm64"]
include:
- arch: amd64
@@ -110,7 +110,7 @@ jobs:
fail-fast: false
matrix:
java-disto: ["ms-openjdk", "amazoncorretto"]
- java-version: ["21", "25"]
+ java-version: ["25"]
name: Create ${{ matrix.java-disto }}-${{ matrix.java-version }} Multi-Arch Manifest
needs: [generate-build-info, package-pinot-docker-image]
steps:
diff --git a/.github/workflows/build-pinot-docker-image.yml b/.github/workflows/build-pinot-docker-image.yml
index 8853651bb68..ab3e1bc44d5 100644
--- a/.github/workflows/build-pinot-docker-image.yml
+++ b/.github/workflows/build-pinot-docker-image.yml
@@ -35,7 +35,7 @@ on:
required: true
jdkVersion:
description: "The JDK version to use for the Pinot build and runtime image."
- default: "21"
+ default: "25"
required: true
tags:
description: "Tags to push of the image, comma separated, e.g. tag1,tag2,tag3"
diff --git a/.github/workflows/nightly-build-all-pinot-docker-image.yml b/.github/workflows/nightly-build-all-pinot-docker-image.yml
index 8abbd810eb8..0aa96070e1a 100644
--- a/.github/workflows/nightly-build-all-pinot-docker-image.yml
+++ b/.github/workflows/nightly-build-all-pinot-docker-image.yml
@@ -47,7 +47,7 @@ jobs:
strategy:
fail-fast: false
matrix:
- java-version: ["21", "25"]
+ java-version: ["25"]
name: Compile JDK-${{ matrix.java-version }} Pinot
needs: [generate-build-info]
steps:
@@ -71,7 +71,7 @@ jobs:
fail-fast: false
matrix:
java-disto: ["ms-openjdk", "amazoncorretto"]
- java-version: ["21", "25"]
+ java-version: ["25"]
arch: ["amd64", "arm64"]
include:
- arch: amd64
@@ -104,7 +104,7 @@ jobs:
fail-fast: false
matrix:
java-disto: ["ms-openjdk", "amazoncorretto"]
- java-version: ["21", "25"]
+ java-version: ["25"]
name: Create ${{ matrix.java-disto }}-${{ matrix.java-version }} Multi-Arch Manifest
needs: [generate-build-info, package-pinot-docker-image]
steps:
diff --git a/.github/workflows/pinot_compatibility_checks.yml b/.github/workflows/pinot_compatibility_checks.yml
index 4ec7a29763c..73e32a5baa1 100644
--- a/.github/workflows/pinot_compatibility_checks.yml
+++ b/.github/workflows/pinot_compatibility_checks.yml
@@ -54,10 +54,10 @@ jobs:
name: Pinot Binary Compatibility Check
steps:
- uses: actions/checkout@v7
- - name: Set up JDK 21
+ - name: Set up JDK 25
uses: actions/setup-java@v5
with:
- java-version: 21
+ java-version: 25
distribution: 'temurin'
cache: 'maven'
# Step that does that actual cache save and restore
@@ -184,10 +184,10 @@ jobs:
name: Pinot Compatibility Regression Testing against ${{ matrix.old_commit }} on ${{ matrix.test_suite }}
steps:
- uses: actions/checkout@v7
- - name: Set up JDK 21
+ - name: Set up JDK 25
uses: actions/setup-java@v5
with:
- java-version: 21
+ java-version: 25
distribution: 'temurin'
cache: 'maven'
- name: Setup node
@@ -243,10 +243,10 @@ jobs:
name: Pinot Multi-Stage Query Engine Compatibility Regression Testing against ${{ matrix.old_commit }} on ${{ matrix.test_suite }}
steps:
- uses: actions/checkout@v7
- - name: Set up JDK 21
+ - name: Set up JDK 25
uses: actions/setup-java@v5
with:
- java-version: 21
+ java-version: 25
distribution: 'temurin'
cache: 'maven'
- name: Setup node
diff --git a/.github/workflows/pinot_compatibility_tests.yml b/.github/workflows/pinot_compatibility_tests.yml
index 47f2bad6293..79d47e1d01c 100644
--- a/.github/workflows/pinot_compatibility_tests.yml
+++ b/.github/workflows/pinot_compatibility_tests.yml
@@ -37,10 +37,10 @@ jobs:
name: Pinot Compatibility Regression Testing against ${{ github.event.inputs.oldCommit }} and ${{ github.event.inputs.newCommit }} on ${{ matrix.test_suite }}
steps:
- uses: actions/checkout@v7
- - name: Set up JDK 21
+ - name: Set up JDK 25
uses: actions/setup-java@v5
with:
- java-version: 21
+ java-version: 25
distribution: 'temurin'
cache: 'maven'
- name: Setup node
diff --git a/.github/workflows/pinot_integration_tests.yml b/.github/workflows/pinot_integration_tests.yml
index 6607b317f43..cf8c7125a77 100644
--- a/.github/workflows/pinot_integration_tests.yml
+++ b/.github/workflows/pinot_integration_tests.yml
@@ -56,7 +56,7 @@ jobs:
fail-fast: false
matrix:
testset: [ 1, 2 ]
- java: [ 21 ]
+ java: [ 25 ]
distribution: [ "temurin" ]
name: Pinot Integration Test Set ${{ matrix.testset }} (${{matrix.distribution}}-${{matrix.java}})
steps:
diff --git a/.github/workflows/pinot_multi_stage_query_engine_compatibility_tests.yml b/.github/workflows/pinot_multi_stage_query_engine_compatibility_tests.yml
index cad4a395a03..ecfbda77b08 100644
--- a/.github/workflows/pinot_multi_stage_query_engine_compatibility_tests.yml
+++ b/.github/workflows/pinot_multi_stage_query_engine_compatibility_tests.yml
@@ -37,10 +37,10 @@ jobs:
name: Pinot Multi-Stage Query Engine Compatibility Regression Testing against ${{ github.event.inputs.oldCommit }} and ${{ github.event.inputs.newCommit }} on ${{ matrix.test_suite }}
steps:
- uses: actions/checkout@v7
- - name: Set up JDK 21
+ - name: Set up JDK 25
uses: actions/setup-java@v5
with:
- java-version: 21
+ java-version: 25
distribution: 'temurin'
cache: 'maven'
- name: Setup node
diff --git a/.github/workflows/pinot_quickstart_tests.yml b/.github/workflows/pinot_quickstart_tests.yml
index 8f303426ae7..e9abd34b598 100644
--- a/.github/workflows/pinot_quickstart_tests.yml
+++ b/.github/workflows/pinot_quickstart_tests.yml
@@ -55,7 +55,7 @@ jobs:
# Changed to false in order to improve coverage using unsafe buffers
fail-fast: false
matrix:
- java: [ 21 ]
+ java: [ 25 ]
distribution: [ "temurin" ]
name: Pinot Quickstart on JDK ${{ matrix.java }}-${{ matrix.distribution }}
steps:
diff --git a/.github/workflows/pinot_unit_tests.yml b/.github/workflows/pinot_unit_tests.yml
index e2ec81d366f..d2b2a8412a0 100644
--- a/.github/workflows/pinot_unit_tests.yml
+++ b/.github/workflows/pinot_unit_tests.yml
@@ -54,10 +54,10 @@ jobs:
name: Pinot Linter Test Set
steps:
- uses: actions/checkout@v7
- - name: Set up JDK 21
+ - name: Set up JDK 25
uses: actions/setup-java@v5
with:
- java-version: 21
+ java-version: 25
distribution: 'temurin'
cache: 'maven'
- name: Install pinot-dependency-verifier into repo
@@ -91,7 +91,7 @@ jobs:
fail-fast: false
matrix:
testset: [ 1, 2 ]
- java: [ 21 ]
+ java: [ 25 ]
distribution: [ "temurin" ]
name: Pinot Unit Test Set ${{ matrix.testset }} (${{matrix.distribution}}-${{matrix.java}})
steps:
diff --git a/.github/workflows/scripts/docker/.pinot_docker_image_build_and_push.sh b/.github/workflows/scripts/docker/.pinot_docker_image_build_and_push.sh
index 6db288f3388..1868d87468e 100755
--- a/.github/workflows/scripts/docker/.pinot_docker_image_build_and_push.sh
+++ b/.github/workflows/scripts/docker/.pinot_docker_image_build_and_push.sh
@@ -28,7 +28,7 @@ if [ -z "${BUILD_PLATFORM}" ]; then
BUILD_PLATFORM="linux/arm64,linux/amd64"
fi
if [ -z "${JDK_VERSION}" ]; then
- JDK_VERSION="21"
+ JDK_VERSION="25"
fi
if [ -z "${PINOT_BASE_IMAGE_TAG}" ]; then
PINOT_BASE_IMAGE_TAG="${JDK_VERSION}-amazoncorretto"
diff --git a/.github/workflows/scripts/docker/.pinot_multi_arch_docker_image_manifest_package.sh b/.github/workflows/scripts/docker/.pinot_multi_arch_docker_image_manifest_package.sh
index a12cd06fbf9..939f46bf825 100755
--- a/.github/workflows/scripts/docker/.pinot_multi_arch_docker_image_manifest_package.sh
+++ b/.github/workflows/scripts/docker/.pinot_multi_arch_docker_image_manifest_package.sh
@@ -49,7 +49,7 @@ for tag in "${tags[@]}"; do
docker manifest push \
${DOCKER_IMAGE_NAME}:${tag}-${baseImageTag}
- if [ "${baseImageTag}" == "21-ms-openjdk" ]; then
+ if [ "${baseImageTag}" == "25-ms-openjdk" ]; then
echo "Creating manifest for tag: ${tag}"
docker manifest create \
${DOCKER_IMAGE_NAME}:${tag} \
diff --git a/.github/workflows/scripts/docker/.pinot_package_single_platform.sh b/.github/workflows/scripts/docker/.pinot_package_single_platform.sh
index 776a666f2ce..30e6f07180e 100755
--- a/.github/workflows/scripts/docker/.pinot_package_single_platform.sh
+++ b/.github/workflows/scripts/docker/.pinot_package_single_platform.sh
@@ -62,9 +62,9 @@ for runtimeImage in "${runtimeImages[@]}"; do
for tag in "${tags[@]}"; do
DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag}-${runtimeImage}-linux-${BUILD_ARCH} "
- # 21-ms-openjdk is the canonical default runtime; promote it as the bare
+ # 25-ms-openjdk is the canonical default runtime; promote it as the bare
# latest-linux-<arch> tag so multi-arch manifests can reference it.
- if [ "${runtimeImage}" == "21-ms-openjdk" ]; then
+ if [ "${runtimeImage}" == "25-ms-openjdk" ]; then
if [ "${tag}" == "latest" ]; then
DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:latest-linux-${BUILD_ARCH} "
fi
@@ -84,7 +84,7 @@ for runtimeImage in "${runtimeImages[@]}"; do
for tag in "${tags[@]}"; do
docker push "${DOCKER_IMAGE_NAME}:${tag}-${runtimeImage}-linux-${BUILD_ARCH}"
- if [ "${runtimeImage}" == "21-ms-openjdk" ]; then
+ if [ "${runtimeImage}" == "25-ms-openjdk" ]; then
if [ "${tag}" == "latest" ]; then
docker push "${DOCKER_IMAGE_NAME}:latest-linux-${BUILD_ARCH}"
fi
diff --git a/CLAUDE.md b/CLAUDE.md
index d9abfc33e77..046bd537a09 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -89,7 +89,7 @@ Apache Pinot is a real-time distributed OLAP datastore for low-latency analytics
- **assembly-descriptor**: Maven assembly descriptor for plugin packaging.
## Build commands
-- **JDK**: Use JDK 21+ for Pinot services and the default build; client and SPI artifacts still target Java 11 bytecode.
+- **JDK**: Use JDK 25+ for Pinot services and the default build; client and SPI artifacts still target Java 11 bytecode.
- **Default build**: `./mvnw clean install`
- **Fast dev build**: `./mvnw verify -Ppinot-fastdev`
- **Full binary/shaded build**: `./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar`
@@ -107,7 +107,7 @@ Apache Pinot is a real-time distributed OLAP datastore for low-latency analytics
## Coding conventions
- Add class-level Javadoc for new classes; describe behavior and thread-safety.
-- Use Javadoc comments (`/** ... */` or `///` syntax); service code targets Java 21 by default.
+- Use Javadoc comments (`/** ... */` or `///` syntax); service code targets Java 25 by default.
- Keep Apache 2.0 license headers on all new source files.
- Preserve backward compatibility across mixed-version broker/server/controller.
- Prefer imports over fully qualified class names (e.g., use `import com.foo.Bar` and refer to `Bar`, not `com.foo.Bar` inline).
diff --git a/LICENSE-binary b/LICENSE-binary
index 9430db9a15f..30034937325 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -425,8 +425,7 @@ org.apache.commons:commons-text:1.15.0
org.apache.curator:curator-client:5.9.0
org.apache.curator:curator-framework:5.9.0
org.apache.curator:curator-recipes:5.9.0
-org.apache.datasketches:datasketches-java:6.2.0
-org.apache.datasketches:datasketches-memory:3.0.2
+org.apache.datasketches:datasketches-java:9.0.0
org.apache.hadoop.thirdparty:hadoop-shaded-guava:1.5.0
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_25:1.5.0
org.apache.hadoop:hadoop-annotations:3.4.3
diff --git a/README.md b/README.md
index a198cbebc46..c8c0aa1266e 100644
--- a/README.md
+++ b/README.md
@@ -124,7 +124,7 @@ Drop your video or a link to your session in the [#pinot-youtube-channel](https:
$ git clone https://github.com/apache/pinot.git
$ cd pinot
-# Pinot services require JDK 21+ to build and run
+# Pinot services require JDK 25+ to build and run
# Java/JDBC clients and SPI artifacts continue to target Java 11 bytecode
# Build Pinot
diff --git a/compatibility-verifier/checkoutAndBuild.sh b/compatibility-verifier/checkoutAndBuild.sh
index ba22c785f79..7e9b928f128 100755
--- a/compatibility-verifier/checkoutAndBuild.sh
+++ b/compatibility-verifier/checkoutAndBuild.sh
@@ -98,7 +98,10 @@ function build() {
local buildTests=$2
local buildId=$3
local buildCompatibilityVerifier=$4
- local versionOption="-Djdk.version=21"
+ # Build each tree with its own pom jdk.version default: old commits may target an
+ # older Java release than the current tree (e.g. release-1.5.0 targets 11, current
+ # targets 25), and javac supports all of them via --release.
+ local versionOption=""
local maxRetry=5
local repoId=${buildId}
diff --git a/docker/images/pinot-base/README.md b/docker/images/pinot-base/README.md
index 8a1bda0f4c9..eb0acdc24c8 100644
--- a/docker/images/pinot-base/README.md
+++ b/docker/images/pinot-base/README.md
@@ -33,22 +33,22 @@ This task can be triggered manually to build the cross platform(amd64 and arm64v
The build shell is:
-For Amazon Corretto 21:
+For Amazon Corretto 25:
```SHELL
-docker buildx build --no-cache --platform=linux/arm64,linux/amd64 --file pinot-base-build/amazoncorretto.dockerfile --tag apachepinot/pinot-base-build:21-amazoncorretto --push .
+docker buildx build --no-cache --platform=linux/arm64,linux/amd64 --file pinot-base-build/amazoncorretto.dockerfile --tag apachepinot/pinot-base-build:25-amazoncorretto --push .
```
```SHELL
-docker buildx build --no-cache --platform=linux/arm64,linux/amd64 --file pinot-base-runtime/amazoncorretto.dockerfile --tag apachepinot/pinot-base-runtime:21-amazoncorretto --push .
+docker buildx build --no-cache --platform=linux/arm64,linux/amd64 --file pinot-base-runtime/amazoncorretto.dockerfile --tag apachepinot/pinot-base-runtime:25-amazoncorretto --push .
```
For MS OpenJDK, the build shell is:
```SHELL
-docker buildx build --no-cache --platform=linux/arm64,linux/amd64 --file pinot-base-build/ms-openjdk.dockerfile --tag apachepinot/pinot-base-build:21-ms-openjdk --push .
+docker buildx build --no-cache --platform=linux/arm64,linux/amd64 --file pinot-base-build/ms-openjdk.dockerfile --tag apachepinot/pinot-base-build:25-ms-openjdk --push .
```
```SHELL
-docker buildx build --no-cache --platform=linux/arm64,linux/amd64 --file pinot-base-runtime/ms-openjdk.dockerfile --tag apachepinot/pinot-base-runtime:21-ms-openjdk --push .
+docker buildx build --no-cache --platform=linux/arm64,linux/amd64 --file pinot-base-runtime/ms-openjdk.dockerfile --tag apachepinot/pinot-base-runtime:25-ms-openjdk --push .
```
diff --git a/docker/images/pinot/Dockerfile b/docker/images/pinot/Dockerfile
index b69061de75b..3d5d857c4a7 100644
--- a/docker/images/pinot/Dockerfile
+++ b/docker/images/pinot/Dockerfile
@@ -16,13 +16,13 @@
# specific language governing permissions and limitations
# under the License.
#
-ARG PINOT_BASE_IMAGE_TAG=21-amazoncorretto
+ARG PINOT_BASE_IMAGE_TAG=25-amazoncorretto
FROM apachepinot/pinot-base-build:${PINOT_BASE_IMAGE_TAG} AS pinot_build_env
LABEL MAINTAINER=dev@pinot.apache.org
ARG PINOT_GIT_REF=master
-ARG JDK_VERSION=21
+ARG JDK_VERSION=25
ARG CI=true
RUN echo "Build Pinot based on image: apachepinot/pinot-base-build:${PINOT_BASE_IMAGE_TAG}"
diff --git a/docker/images/pinot/Dockerfile.build b/docker/images/pinot/Dockerfile.build
index d4b0ccdf5f6..0fe132a4b11 100644
--- a/docker/images/pinot/Dockerfile.build
+++ b/docker/images/pinot/Dockerfile.build
@@ -16,13 +16,13 @@
# specific language governing permissions and limitations
# under the License.
#
-ARG PINOT_BASE_IMAGE_TAG=21-amazoncorretto
+ARG PINOT_BASE_IMAGE_TAG=25-amazoncorretto
FROM apachepinot/pinot-base-build:${PINOT_BASE_IMAGE_TAG} AS pinot_build_env
LABEL MAINTAINER=dev@pinot.apache.org
ARG PINOT_GIT_REF=master
-ARG JDK_VERSION=21
+ARG JDK_VERSION=25
ARG CI=true
RUN echo "Build Pinot based on image: apachepinot/pinot-base-build:${PINOT_BASE_IMAGE_TAG}"
diff --git a/docker/images/pinot/Dockerfile.package b/docker/images/pinot/Dockerfile.package
index 301ba2ed3d6..2390adce39b 100644
--- a/docker/images/pinot/Dockerfile.package
+++ b/docker/images/pinot/Dockerfile.package
@@ -16,7 +16,7 @@
# specific language governing permissions and limitations
# under the License.
#
-ARG PINOT_BUILD_IMAGE_TAG=21-amazoncorretto
+ARG PINOT_BUILD_IMAGE_TAG=25-amazoncorretto
ARG PINOT_RUNTIME_IMAGE_TAG=${PINOT_BUILD_IMAGE_TAG}
FROM --platform=linux/amd64 pinot-build:${PINOT_BUILD_IMAGE_TAG} AS pinot_build_dist
FROM apachepinot/pinot-base-runtime:${PINOT_RUNTIME_IMAGE_TAG}
diff --git a/helm/pinot/values.yaml b/helm/pinot/values.yaml
index 253cf61cb58..735698522ee 100644
--- a/helm/pinot/values.yaml
+++ b/helm/pinot/values.yaml
@@ -28,10 +28,10 @@ image:
# - `latest` tag is always available and points to the nightly pinot master branch build
# - `release-x.y.z` or `x.y.z` tags are available for each release, e.g. release-1.0.0, release-0.12.1, 1.0.0, 0.12.1, etc.
#
- # Default JDK comes with Amazon Corretto 21, here are also images with supported JDK 21 variants:
- # - Amazon Corretto 21, e.g. `latest`, `latest-21`, `1.0.0-21`, `latest-21-amazoncorretto`,
- # `1.0.0-21-amazoncorretto`
- # - MS OpenJDK 21, e.g. `latest-21-ms-openjdk`, `1.0.0-21-ms-openjdk`
+ # Default JDK comes with Amazon Corretto 25, here are also images with supported JDK 25 variants:
+ # - Amazon Corretto 25, e.g. `latest`, `latest-25`, `1.0.0-25`, `latest-25-amazoncorretto`,
+ # `1.0.0-25-amazoncorretto`
+ # - MS OpenJDK 25, e.g. `latest-25-ms-openjdk`, `1.0.0-25-ms-openjdk`
tag: latest # 1.0.0, 0.12.1, latest
pullPolicy: Always # Use IfNotPresent when you pinged a version of image tag
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java b/pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java
index ed611ee6803..279e886778f 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/common/ObjectSerDeUtils.java
@@ -56,6 +56,7 @@ import it.unimi.dsi.fastutil.objects.ObjectLinkedOpenHashSet;
import it.unimi.dsi.fastutil.objects.ObjectOpenHashSet;
import it.unimi.dsi.fastutil.objects.ObjectSet;
import java.io.IOException;
+import java.lang.foreign.MemorySegment;
import java.math.BigDecimal;
import java.nio.ByteBuffer;
import java.nio.IntBuffer;
@@ -67,11 +68,11 @@ import java.util.PriorityQueue;
import java.util.Set;
import org.apache.datasketches.common.ArrayOfStringsSerDe;
import org.apache.datasketches.cpc.CpcSketch;
-import org.apache.datasketches.frequencies.ItemsSketch;
-import org.apache.datasketches.frequencies.LongsSketch;
+import org.apache.datasketches.frequencies.FrequentItemsSketch;
+import org.apache.datasketches.frequencies.FrequentLongsSketch;
import org.apache.datasketches.kll.KllDoublesSketch;
-import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.theta.Sketch;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.tuple.TupleSketch;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.datasketches.tuple.aninteger.IntegerSummaryDeserializer;
import org.apache.pinot.common.CustomObject;
@@ -240,7 +241,7 @@ public class ObjectSerDeUtils {
return ObjectType.IntSet;
} else if (value instanceof TDigest) {
return ObjectType.TDigest;
- } else if (value instanceof Sketch) {
+ } else if (value instanceof ThetaSketch) {
return ObjectType.DataSketch;
} else if (value instanceof KllDoublesSketch) {
return ObjectType.KllDataSketch;
@@ -290,13 +291,13 @@ public class ObjectSerDeUtils {
return ObjectType.VarianceTuple;
} else if (value instanceof PinotFourthMoment) {
return ObjectType.PinotFourthMoment;
- } else if (value instanceof org.apache.datasketches.tuple.Sketch) {
+ } else if (value instanceof TupleSketch) {
return ObjectType.IntegerTupleSketch;
} else if (value instanceof ExprMinMaxObject) {
return ObjectType.ExprMinMaxObject;
- } else if (value instanceof ItemsSketch) {
+ } else if (value instanceof FrequentItemsSketch) {
return ObjectType.FrequentStringsSketch;
- } else if (value instanceof LongsSketch) {
+ } else if (value instanceof FrequentLongsSketch) {
return ObjectType.FrequentLongsSketch;
} else if (value instanceof HyperLogLogPlus) {
return ObjectType.HyperLogLogPlus;
@@ -1139,10 +1140,10 @@ public class ObjectSerDeUtils {
}
};
- public static final ObjectSerDe<Sketch> DATA_SKETCH_THETA_SER_DE = new ObjectSerDe<Sketch>() {
+ public static final ObjectSerDe<ThetaSketch> DATA_SKETCH_THETA_SER_DE = new ObjectSerDe<ThetaSketch>() {
@Override
- public byte[] serialize(Sketch value) {
+ public byte[] serialize(ThetaSketch value) {
// The serializer should respect existing ordering to enable "early stop"
// optimisations on unions.
if (!value.isCompact()) {
@@ -1152,36 +1153,36 @@ public class ObjectSerDeUtils {
}
@Override
- public Sketch deserialize(byte[] bytes) {
- return Sketch.wrap(Memory.wrap(bytes));
+ public ThetaSketch deserialize(byte[] bytes) {
+ return ThetaSketch.wrap(MemorySegment.ofArray(bytes).asReadOnly());
}
@Override
- public Sketch deserialize(ByteBuffer byteBuffer) {
+ public ThetaSketch deserialize(ByteBuffer byteBuffer) {
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes);
- return Sketch.wrap(Memory.wrap(bytes));
+ return ThetaSketch.wrap(MemorySegment.ofArray(bytes).asReadOnly());
}
};
- public static final ObjectSerDe<org.apache.datasketches.tuple.Sketch<IntegerSummary>> DATA_SKETCH_INT_TUPLE_SER_DE =
- new ObjectSerDe<org.apache.datasketches.tuple.Sketch<IntegerSummary>>() {
+ public static final ObjectSerDe<TupleSketch<IntegerSummary>> DATA_SKETCH_INT_TUPLE_SER_DE =
+ new ObjectSerDe<TupleSketch<IntegerSummary>>() {
@Override
- public byte[] serialize(org.apache.datasketches.tuple.Sketch<IntegerSummary> value) {
+ public byte[] serialize(TupleSketch<IntegerSummary> value) {
return value.compact().toByteArray();
}
@Override
- public org.apache.datasketches.tuple.Sketch<IntegerSummary> deserialize(byte[] bytes) {
- return org.apache.datasketches.tuple.Sketches.heapifySketch(Memory.wrap(bytes),
+ public TupleSketch<IntegerSummary> deserialize(byte[] bytes) {
+ return TupleSketch.heapifySketch(MemorySegment.ofArray(bytes),
new IntegerSummaryDeserializer());
}
@Override
- public org.apache.datasketches.tuple.Sketch<IntegerSummary> deserialize(ByteBuffer byteBuffer) {
+ public TupleSketch<IntegerSummary> deserialize(ByteBuffer byteBuffer) {
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes);
- return org.apache.datasketches.tuple.Sketches.heapifySketch(Memory.wrap(bytes),
+ return TupleSketch.heapifySketch(MemorySegment.ofArray(bytes),
new IntegerSummaryDeserializer());
}
};
@@ -1195,14 +1196,14 @@ public class ObjectSerDeUtils {
@Override
public KllDoublesSketch deserialize(byte[] bytes) {
- return KllDoublesSketch.wrap(Memory.wrap(bytes));
+ return KllDoublesSketch.wrap(MemorySegment.ofArray(bytes).asReadOnly());
}
@Override
public KllDoublesSketch deserialize(ByteBuffer byteBuffer) {
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes);
- return KllDoublesSketch.wrap(Memory.wrap(bytes));
+ return KllDoublesSketch.wrap(MemorySegment.ofArray(bytes).asReadOnly());
}
};
@@ -1214,14 +1215,14 @@ public class ObjectSerDeUtils {
@Override
public CpcSketch deserialize(byte[] bytes) {
- return CpcSketch.heapify(Memory.wrap(bytes));
+ return CpcSketch.heapify(MemorySegment.ofArray(bytes));
}
@Override
public CpcSketch deserialize(ByteBuffer byteBuffer) {
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes);
- return CpcSketch.heapify(Memory.wrap(bytes));
+ return CpcSketch.heapify(MemorySegment.ofArray(bytes));
}
};
@@ -1654,43 +1655,43 @@ public class ObjectSerDeUtils {
}
};
- public static final ObjectSerDe<ItemsSketch<String>> FREQUENT_STRINGS_SKETCH_SER_DE =
+ public static final ObjectSerDe<FrequentItemsSketch<String>> FREQUENT_STRINGS_SKETCH_SER_DE =
new ObjectSerDe<>() {
@Override
- public byte[] serialize(ItemsSketch<String> sketch) {
+ public byte[] serialize(FrequentItemsSketch<String> sketch) {
return sketch.toByteArray(new ArrayOfStringsSerDe());
}
@Override
- public ItemsSketch<String> deserialize(byte[] bytes) {
- return ItemsSketch.getInstance(Memory.wrap(bytes), new ArrayOfStringsSerDe());
+ public FrequentItemsSketch<String> deserialize(byte[] bytes) {
+ return FrequentItemsSketch.getInstance(MemorySegment.ofArray(bytes), new ArrayOfStringsSerDe());
}
@Override
- public ItemsSketch<String> deserialize(ByteBuffer byteBuffer) {
+ public FrequentItemsSketch<String> deserialize(ByteBuffer byteBuffer) {
byte[] arr = new byte[byteBuffer.remaining()];
byteBuffer.get(arr);
- return ItemsSketch.getInstance(Memory.wrap(arr), new ArrayOfStringsSerDe());
+ return FrequentItemsSketch.getInstance(MemorySegment.ofArray(arr), new ArrayOfStringsSerDe());
}
};
- public static final ObjectSerDe<LongsSketch> FREQUENT_LONGS_SKETCH_SER_DE =
+ public static final ObjectSerDe<FrequentLongsSketch> FREQUENT_LONGS_SKETCH_SER_DE =
new ObjectSerDe<>() {
@Override
- public byte[] serialize(LongsSketch sketch) {
+ public byte[] serialize(FrequentLongsSketch sketch) {
return sketch.toByteArray();
}
@Override
- public LongsSketch deserialize(byte[] bytes) {
- return LongsSketch.getInstance(Memory.wrap(bytes));
+ public FrequentLongsSketch deserialize(byte[] bytes) {
+ return FrequentLongsSketch.getInstance(MemorySegment.ofArray(bytes));
}
@Override
- public LongsSketch deserialize(ByteBuffer byteBuffer) {
+ public FrequentLongsSketch deserialize(ByteBuffer byteBuffer) {
byte[] arr = new byte[byteBuffer.remaining()];
byteBuffer.get(arr);
- return LongsSketch.getInstance(Memory.wrap(arr));
+ return FrequentLongsSketch.getInstance(MemorySegment.ofArray(arr));
}
};
@@ -1723,7 +1724,7 @@ public class ObjectSerDeUtils {
@Override
public byte[] serialize(ThetaSketchAccumulator thetaSketchBuffer) {
- Sketch sketch = thetaSketchBuffer.getResult();
+ ThetaSketch sketch = thetaSketchBuffer.getResult();
return sketch.toByteArray();
}
@@ -1739,7 +1740,7 @@ public class ObjectSerDeUtils {
ThetaSketchAccumulator thetaSketchAccumulator = new ThetaSketchAccumulator();
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes);
- Sketch sketch = Sketch.wrap(Memory.wrap(bytes));
+ ThetaSketch sketch = ThetaSketch.wrap(MemorySegment.ofArray(bytes).asReadOnly());
thetaSketchAccumulator.apply(sketch);
return thetaSketchAccumulator;
}
@@ -1750,7 +1751,7 @@ public class ObjectSerDeUtils {
@Override
public byte[] serialize(TupleIntSketchAccumulator tupleIntSketchBuffer) {
- org.apache.datasketches.tuple.Sketch<IntegerSummary> sketch = tupleIntSketchBuffer.getResult();
+ TupleSketch<IntegerSummary> sketch = tupleIntSketchBuffer.getResult();
return sketch.toByteArray();
}
@@ -1766,8 +1767,8 @@ public class ObjectSerDeUtils {
TupleIntSketchAccumulator tupleIntSketchAccumulator = new TupleIntSketchAccumulator();
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes);
- org.apache.datasketches.tuple.Sketch<IntegerSummary> sketch =
- org.apache.datasketches.tuple.Sketches.heapifySketch(Memory.wrap(bytes),
+ TupleSketch<IntegerSummary> sketch =
+ TupleSketch.heapifySketch(MemorySegment.ofArray(bytes),
new IntegerSummaryDeserializer());
tupleIntSketchAccumulator.apply(sketch);
return tupleIntSketchAccumulator;
@@ -1795,7 +1796,7 @@ public class ObjectSerDeUtils {
CpcSketchAccumulator cpcSketchAccumulator = new CpcSketchAccumulator();
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes);
- CpcSketch sketch = CpcSketch.heapify(Memory.wrap(bytes));
+ CpcSketch sketch = CpcSketch.heapify(MemorySegment.ofArray(bytes));
cpcSketchAccumulator.apply(sketch);
return cpcSketchAccumulator;
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/function/scalar/SketchFunctions.java b/pinot-core/src/main/java/org/apache/pinot/core/function/scalar/SketchFunctions.java
index a3b4b65f021..84ad27cfb35 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/function/scalar/SketchFunctions.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/function/scalar/SketchFunctions.java
@@ -20,22 +20,25 @@ package org.apache.pinot.core.function.scalar;
import com.clearspring.analytics.stream.cardinality.HyperLogLog;
import com.dynatrace.hash4j.distinctcount.UltraLogLog;
+import java.lang.foreign.MemorySegment;
import java.math.BigDecimal;
import java.util.Base64;
import javax.annotation.Nullable;
import org.apache.datasketches.cpc.CpcSketch;
import org.apache.datasketches.cpc.CpcUnion;
-import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.theta.AnotB;
-import org.apache.datasketches.theta.Intersection;
-import org.apache.datasketches.theta.SetOperationBuilder;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.Sketches;
-import org.apache.datasketches.theta.Union;
-import org.apache.datasketches.theta.UpdateSketch;
-import org.apache.datasketches.tuple.aninteger.IntegerSketch;
+import org.apache.datasketches.theta.ThetaAnotB;
+import org.apache.datasketches.theta.ThetaIntersection;
+import org.apache.datasketches.theta.ThetaSetOperationBuilder;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.ThetaUnion;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
+import org.apache.datasketches.tuple.TupleAnotB;
+import org.apache.datasketches.tuple.TupleIntersection;
+import org.apache.datasketches.tuple.TupleSketch;
+import org.apache.datasketches.tuple.TupleUnion;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.datasketches.tuple.aninteger.IntegerSummarySetOperations;
+import org.apache.datasketches.tuple.aninteger.IntegerTupleSketch;
import org.apache.pinot.core.common.ObjectSerDeUtils;
import org.apache.pinot.segment.local.utils.UltraLogLogUtils;
import org.apache.pinot.spi.annotations.ScalarFunction;
@@ -85,7 +88,7 @@ import org.apache.pinot.spi.utils.CommonConstants;
* }
*/
public class SketchFunctions {
- private static final SetOperationBuilder SET_OPERATION_BUILDER = new SetOperationBuilder();
+ private static final ThetaSetOperationBuilder SET_OPERATION_BUILDER = new ThetaSetOperationBuilder();
private SketchFunctions() {
}
@@ -110,7 +113,7 @@ public class SketchFunctions {
*/
@ScalarFunction(nullableParameters = true)
public static byte[] toThetaSketch(@Nullable Object input, int nominalEntries) {
- UpdateSketch sketch = Sketches.updateSketchBuilder().setNominalEntries(nominalEntries).build();
+ UpdatableThetaSketch sketch = UpdatableThetaSketch.builder().setNominalEntries(nominalEntries).build();
if (input != null) {
if (input instanceof Integer) {
sketch.update((Integer) input);
@@ -185,7 +188,7 @@ public class SketchFunctions {
*/
@ScalarFunction(nullableParameters = true)
public static byte[] toIntegerSumTupleSketch(@Nullable Object key, Integer value, int lgK) {
- IntegerSketch is = new IntegerSketch(lgK, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch is = new IntegerTupleSketch(lgK, IntegerSummary.Mode.Sum);
if (value != null && key != null) {
if (key instanceof Integer) {
is.update((Integer) key, value);
@@ -214,48 +217,48 @@ public class SketchFunctions {
}
@ScalarFunction
- public static Sketch thetaSketchUnion(Object o1, Object o2) {
+ public static ThetaSketch thetaSketchUnion(Object o1, Object o2) {
return thetaSketchUnionVar(o1, o2);
}
@ScalarFunction
- public static Sketch thetaSketchUnion(Object o1, Object o2, Object o3) {
+ public static ThetaSketch thetaSketchUnion(Object o1, Object o2, Object o3) {
return thetaSketchUnionVar(o1, o2, o3);
}
@ScalarFunction
- public static Sketch thetaSketchUnion(Object o1, Object o2, Object o3, Object o4) {
+ public static ThetaSketch thetaSketchUnion(Object o1, Object o2, Object o3, Object o4) {
return thetaSketchUnionVar(o1, o2, o3, o4);
}
@ScalarFunction
- public static Sketch thetaSketchUnion(Object o1, Object o2, Object o3, Object o4, Object o5) {
+ public static ThetaSketch thetaSketchUnion(Object o1, Object o2, Object o3, Object o4, Object o5) {
return thetaSketchUnionVar(o1, o2, o3, o4, o5);
}
@ScalarFunction
- public static Sketch thetaSketchIntersect(Object o1, Object o2) {
+ public static ThetaSketch thetaSketchIntersect(Object o1, Object o2) {
return thetaSketchIntersectVar(o1, o2);
}
@ScalarFunction
- public static Sketch thetaSketchIntersect(Object o1, Object o2, Object o3) {
+ public static ThetaSketch thetaSketchIntersect(Object o1, Object o2, Object o3) {
return thetaSketchIntersectVar(o1, o2, o3);
}
@ScalarFunction
- public static Sketch thetaSketchIntersect(Object o1, Object o2, Object o3, Object o4) {
+ public static ThetaSketch thetaSketchIntersect(Object o1, Object o2, Object o3, Object o4) {
return thetaSketchIntersectVar(o1, o2, o3, o4);
}
@ScalarFunction
- public static Sketch thetaSketchIntersect(Object o1, Object o2, Object o3, Object o4, Object o5) {
+ public static ThetaSketch thetaSketchIntersect(Object o1, Object o2, Object o3, Object o4, Object o5) {
return thetaSketchIntersectVar(o1, o2, o3, o4, o5);
}
@ScalarFunction
- public static Sketch thetaSketchDiff(Object sketchObjectA, Object sketchObjectB) {
- AnotB diff = SET_OPERATION_BUILDER.buildANotB();
+ public static ThetaSketch thetaSketchDiff(Object sketchObjectA, Object sketchObjectB) {
+ ThetaAnotB diff = SET_OPERATION_BUILDER.buildANotB();
diff.setA(asThetaSketch(sketchObjectA));
diff.notB(asThetaSketch(sketchObjectB));
return diff.getResult(false, null, false);
@@ -266,30 +269,30 @@ public class SketchFunctions {
return asThetaSketch(sketchObject).toString();
}
- private static Sketch thetaSketchUnionVar(Object... sketchObjects) {
- Union union = SET_OPERATION_BUILDER.buildUnion();
+ private static ThetaSketch thetaSketchUnionVar(Object... sketchObjects) {
+ ThetaUnion union = SET_OPERATION_BUILDER.buildUnion();
for (Object sketchObj : sketchObjects) {
union.union(asThetaSketch(sketchObj));
}
return union.getResult(false, null);
}
- private static Sketch thetaSketchIntersectVar(Object... sketchObjects) {
- Intersection intersection = SET_OPERATION_BUILDER.buildIntersection();
+ private static ThetaSketch thetaSketchIntersectVar(Object... sketchObjects) {
+ ThetaIntersection intersection = SET_OPERATION_BUILDER.buildIntersection();
for (Object sketchObj : sketchObjects) {
intersection.intersect(asThetaSketch(sketchObj));
}
return intersection.getResult(false, null);
}
- private static Sketch asThetaSketch(Object sketchObj) {
+ private static ThetaSketch asThetaSketch(Object sketchObj) {
if (sketchObj instanceof String) {
byte[] decoded = Base64.getDecoder().decode((String) sketchObj);
- return Sketches.wrapSketch(Memory.wrap((decoded)));
- } else if (sketchObj instanceof Sketch) {
- return (Sketch) sketchObj;
+ return ThetaSketch.wrap(MemorySegment.ofArray(decoded).asReadOnly());
+ } else if (sketchObj instanceof ThetaSketch) {
+ return (ThetaSketch) sketchObj;
} else if (sketchObj instanceof byte[]) {
- return Sketches.wrapSketch(Memory.wrap((byte[]) sketchObj));
+ return ThetaSketch.wrap(MemorySegment.ofArray((byte[]) sketchObj).asReadOnly());
} else {
throw new RuntimeException(
"Exception occurred getting estimate from Theta Sketch, unsupported Object type: " + sketchObj.getClass());
@@ -327,8 +330,8 @@ public class SketchFunctions {
}
private static byte[] intTupleSketchUnionVar(IntegerSummary.Mode mode, int nomEntries, Object... sketchObjects) {
- org.apache.datasketches.tuple.Union<IntegerSummary> union =
- new org.apache.datasketches.tuple.Union<>(nomEntries, new IntegerSummarySetOperations(mode, mode));
+ TupleUnion<IntegerSummary> union =
+ new TupleUnion<>(nomEntries, new IntegerSummarySetOperations(mode, mode));
for (Object sketchObj : sketchObjects) {
union.union(asIntegerSketch(sketchObj));
}
@@ -351,8 +354,8 @@ public class SketchFunctions {
}
private static byte[] intTupleSketchIntersectVar(IntegerSummary.Mode mode, Object... sketchObjects) {
- org.apache.datasketches.tuple.Intersection<IntegerSummary> intersection =
- new org.apache.datasketches.tuple.Intersection<>(new IntegerSummarySetOperations(mode, mode));
+ TupleIntersection<IntegerSummary> intersection =
+ new TupleIntersection<>(new IntegerSummarySetOperations(mode, mode));
for (Object sketchObj : sketchObjects) {
intersection.intersect(asIntegerSketch(sketchObj));
}
@@ -361,18 +364,18 @@ public class SketchFunctions {
@ScalarFunction
public static byte[] intSumTupleSketchDiff(Object o1, Object o2) {
- org.apache.datasketches.tuple.AnotB<IntegerSummary> diff = new org.apache.datasketches.tuple.AnotB<>();
+ TupleAnotB<IntegerSummary> diff = new TupleAnotB<>();
diff.setA(asIntegerSketch(o1));
diff.notB(asIntegerSketch(o2));
return ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.serialize(diff.getResult(false).compact());
}
- private static org.apache.datasketches.tuple.Sketch<IntegerSummary> asIntegerSketch(Object sketchObj) {
+ private static TupleSketch<IntegerSummary> asIntegerSketch(Object sketchObj) {
if (sketchObj instanceof String) {
byte[] decoded = Base64.getDecoder().decode((String) sketchObj);
return ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(decoded);
- } else if (sketchObj instanceof org.apache.datasketches.tuple.Sketch) {
- return (org.apache.datasketches.tuple.Sketch<IntegerSummary>) sketchObj;
+ } else if (sketchObj instanceof TupleSketch) {
+ return (TupleSketch<IntegerSummary>) sketchObj;
} else if (sketchObj instanceof byte[]) {
return ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize((byte[]) sketchObj);
} else {
@@ -464,10 +467,10 @@ public class SketchFunctions {
if (sketchObj instanceof CpcSketch) {
return (CpcSketch) sketchObj;
} else if (sketchObj instanceof byte[]) {
- return CpcSketch.heapify(Memory.wrap((byte[]) sketchObj));
+ return CpcSketch.heapify(MemorySegment.ofArray((byte[]) sketchObj));
} else if (sketchObj instanceof String) {
byte[] decoded = Base64.getDecoder().decode((String) sketchObj);
- return CpcSketch.heapify(Memory.wrap((decoded)));
+ return CpcSketch.heapify(MemorySegment.ofArray(decoded));
} else {
throw new RuntimeException(
"Exception occurred getting estimate from CPC Sketch, unsupported Object type: " + sketchObj.getClass());
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AvgValueIntegerTupleSketchAggregationFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AvgValueIntegerTupleSketchAggregationFunction.java
index 16fd6751b2d..27811d400fe 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AvgValueIntegerTupleSketchAggregationFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/AvgValueIntegerTupleSketchAggregationFunction.java
@@ -19,7 +19,7 @@
package org.apache.pinot.core.query.aggregation.function;
import java.util.List;
-import org.apache.datasketches.tuple.Sketch;
+import org.apache.datasketches.tuple.TupleSketch;
import org.apache.datasketches.tuple.TupleSketchIterator;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.pinot.common.request.context.ExpressionContext;
@@ -52,7 +52,7 @@ public class AvgValueIntegerTupleSketchAggregationFunction
accumulator.setNominalEntries(_nominalEntries);
accumulator.setSetOperations(_setOps);
accumulator.setThreshold(_accumulatorThreshold);
- Sketch<IntegerSummary> result = accumulator.getResult();
+ TupleSketch<IntegerSummary> result = accumulator.getResult();
if (result.isEmpty() || result.getRetainedEntries() == 0) {
// there is nothing to average, return null
return null;
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountCPCSketchAggregationFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountCPCSketchAggregationFunction.java
index fd2b40395f8..4b11e09ba3c 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountCPCSketchAggregationFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountCPCSketchAggregationFunction.java
@@ -19,12 +19,12 @@
package org.apache.pinot.core.query.aggregation.function;
import com.google.common.base.Preconditions;
+import java.lang.foreign.MemorySegment;
import java.util.List;
import java.util.Map;
import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
import org.apache.datasketches.cpc.CpcSketch;
-import org.apache.datasketches.memory.Memory;
import org.apache.pinot.common.CustomObject;
import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.common.utils.DataSchema;
@@ -614,7 +614,7 @@ public class DistinctCountCPCSketchAggregationFunction
CpcSketch[] sketches = new CpcSketch[length];
for (int i = 0; i < length; i++) {
byte[] bytes = serializedSketches[i];
- sketches[i] = bytes.length > 0 ? CpcSketch.heapify(Memory.wrap(bytes)) : null;
+ sketches[i] = bytes.length > 0 ? CpcSketch.heapify(MemorySegment.ofArray(bytes)) : null;
}
return sketches;
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountRawThetaSketchAggregationFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountRawThetaSketchAggregationFunction.java
index eb54e91287a..b1c524f9f3c 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountRawThetaSketchAggregationFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountRawThetaSketchAggregationFunction.java
@@ -22,7 +22,7 @@ import java.util.ArrayList;
import java.util.Base64;
import java.util.List;
import javax.annotation.Nullable;
-import org.apache.datasketches.theta.Sketch;
+import org.apache.datasketches.theta.ThetaSketch;
import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.common.utils.DataSchema.ColumnDataType;
import org.apache.pinot.segment.local.customobject.ThetaSketchAccumulator;
@@ -56,7 +56,7 @@ public class DistinctCountRawThetaSketchAggregationFunction extends DistinctCoun
return null;
}
int numAccumulators = accumulators.size();
- List<Sketch> mergedSketches = new ArrayList<>(numAccumulators);
+ List<ThetaSketch> mergedSketches = new ArrayList<>(numAccumulators);
for (Object object : accumulators) {
ThetaSketchAccumulator accumulator = convertSketchAccumulator(object);
@@ -65,7 +65,7 @@ public class DistinctCountRawThetaSketchAggregationFunction extends DistinctCoun
mergedSketches.add(accumulator.getResult());
}
- Sketch sketch = evaluatePostAggregationExpression(mergedSketches);
+ ThetaSketch sketch = evaluatePostAggregationExpression(mergedSketches);
return Base64.getEncoder().encodeToString(sketch.toByteArray());
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java
index 69edf894c45..c61c2ef90ec 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/DistinctCountThetaSketchAggregationFunction.java
@@ -19,6 +19,7 @@
package org.apache.pinot.core.query.aggregation.function;
import com.google.common.base.Preconditions;
+import java.lang.foreign.MemorySegment;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.HashMap;
@@ -26,14 +27,13 @@ import java.util.List;
import java.util.Map;
import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
-import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.theta.AnotB;
-import org.apache.datasketches.theta.Intersection;
-import org.apache.datasketches.theta.SetOperationBuilder;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.Union;
-import org.apache.datasketches.theta.UpdateSketch;
-import org.apache.datasketches.theta.UpdateSketchBuilder;
+import org.apache.datasketches.theta.ThetaAnotB;
+import org.apache.datasketches.theta.ThetaIntersection;
+import org.apache.datasketches.theta.ThetaSetOperationBuilder;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.ThetaUnion;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketchBuilder;
import org.apache.datasketches.thetacommon.ThetaUtil;
import org.apache.pinot.common.CustomObject;
import org.apache.pinot.common.request.Expression;
@@ -96,15 +96,15 @@ public class DistinctCountThetaSketchAggregationFunction
private final boolean _includeDefaultSketch;
private final List<FilterEvaluator> _filterEvaluators;
private final ExpressionContext _postAggregationExpression;
- private final UpdateSketchBuilder _updateSketchBuilder = new UpdateSketchBuilder();
+ private final UpdatableThetaSketchBuilder _updateSketchBuilder = new UpdatableThetaSketchBuilder();
private int _nominalEntries = ThetaUtil.DEFAULT_NOMINAL_ENTRIES;
- protected final SetOperationBuilder _setOperationBuilder = new SetOperationBuilder();
+ protected final ThetaSetOperationBuilder _setOperationBuilder = new ThetaSetOperationBuilder();
protected int _accumulatorThreshold = DEFAULT_ACCUMULATOR_THRESHOLD;
public DistinctCountThetaSketchAggregationFunction(List<ExpressionContext> arguments) {
super(arguments.get(0));
- // Initialize the UpdateSketchBuilder and SetOperationBuilder with the parameters
+ // Initialize the UpdatableThetaSketchBuilder and ThetaSetOperationBuilder with the parameters
int numArguments = arguments.size();
if (numArguments > 1) {
ExpressionContext paramsExpression = arguments.get(1);
@@ -200,20 +200,20 @@ public class DistinctCountThetaSketchAggregationFunction
// Main expression is always index 0
if (valueTypes[0] != DataType.BYTES) {
- List<UpdateSketch> updateSketches = getUpdateSketches(aggregationResultHolder);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(aggregationResultHolder);
if (singleValues[0]) {
switch (valueTypes[0]) {
case INT:
int[] intValues = (int[]) valueArrays[0];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int i = 0; i < length; i++) {
defaultSketch.update(intValues[i]);
}
}
for (int i = 0; i < numFilters; i++) {
FilterEvaluator filterEvaluator = _filterEvaluators.get(i);
- UpdateSketch updateSketch = updateSketches.get(i + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(i + 1);
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
updateSketch.update(intValues[j]);
@@ -224,14 +224,14 @@ public class DistinctCountThetaSketchAggregationFunction
case LONG:
long[] longValues = (long[]) valueArrays[0];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int i = 0; i < length; i++) {
defaultSketch.update(longValues[i]);
}
}
for (int i = 0; i < numFilters; i++) {
FilterEvaluator filterEvaluator = _filterEvaluators.get(i);
- UpdateSketch updateSketch = updateSketches.get(i + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(i + 1);
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
updateSketch.update(longValues[j]);
@@ -242,14 +242,14 @@ public class DistinctCountThetaSketchAggregationFunction
case FLOAT:
float[] floatValues = (float[]) valueArrays[0];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int i = 0; i < length; i++) {
defaultSketch.update(floatValues[i]);
}
}
for (int i = 0; i < numFilters; i++) {
FilterEvaluator filterEvaluator = _filterEvaluators.get(i);
- UpdateSketch updateSketch = updateSketches.get(i + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(i + 1);
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
updateSketch.update(floatValues[j]);
@@ -260,14 +260,14 @@ public class DistinctCountThetaSketchAggregationFunction
case DOUBLE:
double[] doubleValues = (double[]) valueArrays[0];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int i = 0; i < length; i++) {
defaultSketch.update(doubleValues[i]);
}
}
for (int i = 0; i < numFilters; i++) {
FilterEvaluator filterEvaluator = _filterEvaluators.get(i);
- UpdateSketch updateSketch = updateSketches.get(i + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(i + 1);
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
updateSketch.update(doubleValues[j]);
@@ -278,14 +278,14 @@ public class DistinctCountThetaSketchAggregationFunction
case STRING:
String[] stringValues = (String[]) valueArrays[0];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int i = 0; i < length; i++) {
defaultSketch.update(stringValues[i]);
}
}
for (int i = 0; i < numFilters; i++) {
FilterEvaluator filterEvaluator = _filterEvaluators.get(i);
- UpdateSketch updateSketch = updateSketches.get(i + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(i + 1);
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
updateSketch.update(stringValues[j]);
@@ -303,7 +303,7 @@ public class DistinctCountThetaSketchAggregationFunction
case INT:
int[][] intValues = (int[][]) valueArrays[0];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int i = 0; i < length; i++) {
for (int value : intValues[i]) {
defaultSketch.update(value);
@@ -312,7 +312,7 @@ public class DistinctCountThetaSketchAggregationFunction
}
for (int i = 0; i < numFilters; i++) {
FilterEvaluator filterEvaluator = _filterEvaluators.get(i);
- UpdateSketch updateSketch = updateSketches.get(i + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(i + 1);
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
for (int value : intValues[j]) {
@@ -325,7 +325,7 @@ public class DistinctCountThetaSketchAggregationFunction
case LONG:
long[][] longValues = (long[][]) valueArrays[0];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int i = 0; i < length; i++) {
for (long value : longValues[i]) {
defaultSketch.update(value);
@@ -334,7 +334,7 @@ public class DistinctCountThetaSketchAggregationFunction
}
for (int i = 0; i < numFilters; i++) {
FilterEvaluator filterEvaluator = _filterEvaluators.get(i);
- UpdateSketch updateSketch = updateSketches.get(i + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(i + 1);
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
for (long value : longValues[j]) {
@@ -347,7 +347,7 @@ public class DistinctCountThetaSketchAggregationFunction
case FLOAT:
float[][] floatValues = (float[][]) valueArrays[0];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int i = 0; i < length; i++) {
for (float value : floatValues[i]) {
defaultSketch.update(value);
@@ -356,7 +356,7 @@ public class DistinctCountThetaSketchAggregationFunction
}
for (int i = 0; i < numFilters; i++) {
FilterEvaluator filterEvaluator = _filterEvaluators.get(i);
- UpdateSketch updateSketch = updateSketches.get(i + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(i + 1);
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
for (float value : floatValues[j]) {
@@ -369,7 +369,7 @@ public class DistinctCountThetaSketchAggregationFunction
case DOUBLE:
double[][] doubleValues = (double[][]) valueArrays[0];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int i = 0; i < length; i++) {
for (double value : doubleValues[i]) {
defaultSketch.update(value);
@@ -378,7 +378,7 @@ public class DistinctCountThetaSketchAggregationFunction
}
for (int i = 0; i < numFilters; i++) {
FilterEvaluator filterEvaluator = _filterEvaluators.get(i);
- UpdateSketch updateSketch = updateSketches.get(i + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(i + 1);
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
for (double value : doubleValues[j]) {
@@ -391,7 +391,7 @@ public class DistinctCountThetaSketchAggregationFunction
case STRING:
String[][] stringValues = (String[][]) valueArrays[0];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int i = 0; i < length; i++) {
for (String value : stringValues[i]) {
defaultSketch.update(value);
@@ -400,7 +400,7 @@ public class DistinctCountThetaSketchAggregationFunction
}
for (int i = 0; i < numFilters; i++) {
FilterEvaluator filterEvaluator = _filterEvaluators.get(i);
- UpdateSketch updateSketch = updateSketches.get(i + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(i + 1);
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
for (String value : stringValues[j]) {
@@ -418,10 +418,10 @@ public class DistinctCountThetaSketchAggregationFunction
} else {
// Serialized sketch
List<ThetaSketchAccumulator> thetaSketchAccumulators = getUnions(aggregationResultHolder);
- Sketch[] sketches = deserializeSketches((byte[][]) valueArrays[0], length);
+ ThetaSketch[] sketches = deserializeSketches((byte[][]) valueArrays[0], length);
if (_includeDefaultSketch) {
ThetaSketchAccumulator defaultThetaAccumulator = thetaSketchAccumulators.get(0);
- for (Sketch sketch : sketches) {
+ for (ThetaSketch sketch : sketches) {
defaultThetaAccumulator.apply(sketch);
}
}
@@ -454,7 +454,7 @@ public class DistinctCountThetaSketchAggregationFunction
case INT:
int[] intValues = (int[]) valueArrays[0];
for (int i = 0; i < length; i++) {
- List<UpdateSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
int value = intValues[i];
if (_includeDefaultSketch) {
updateSketches.get(0).update(value);
@@ -469,7 +469,7 @@ public class DistinctCountThetaSketchAggregationFunction
case LONG:
long[] longValues = (long[]) valueArrays[0];
for (int i = 0; i < length; i++) {
- List<UpdateSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
long value = longValues[i];
if (_includeDefaultSketch) {
updateSketches.get(0).update(value);
@@ -484,7 +484,7 @@ public class DistinctCountThetaSketchAggregationFunction
case FLOAT:
float[] floatValues = (float[]) valueArrays[0];
for (int i = 0; i < length; i++) {
- List<UpdateSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
float value = floatValues[i];
if (_includeDefaultSketch) {
updateSketches.get(0).update(value);
@@ -499,7 +499,7 @@ public class DistinctCountThetaSketchAggregationFunction
case DOUBLE:
double[] doubleValues = (double[]) valueArrays[0];
for (int i = 0; i < length; i++) {
- List<UpdateSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
double value = doubleValues[i];
if (_includeDefaultSketch) {
updateSketches.get(0).update(value);
@@ -514,7 +514,7 @@ public class DistinctCountThetaSketchAggregationFunction
case STRING:
String[] stringValues = (String[]) valueArrays[0];
for (int i = 0; i < length; i++) {
- List<UpdateSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
String value = stringValues[i];
if (_includeDefaultSketch) {
updateSketches.get(0).update(value);
@@ -536,17 +536,17 @@ public class DistinctCountThetaSketchAggregationFunction
case INT:
int[][] intValues = (int[][]) valueArrays[0];
for (int i = 0; i < length; i++) {
- List<UpdateSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
int[] values = intValues[i];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (int value : values) {
defaultSketch.update(value);
}
}
for (int j = 0; j < numFilters; j++) {
if (_filterEvaluators.get(j).evaluate(singleValues, valueTypes, valueArrays, i)) {
- UpdateSketch updateSketch = updateSketches.get(j + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(j + 1);
for (int value : values) {
updateSketch.update(value);
}
@@ -557,17 +557,17 @@ public class DistinctCountThetaSketchAggregationFunction
case LONG:
long[][] longValues = (long[][]) valueArrays[0];
for (int i = 0; i < length; i++) {
- List<UpdateSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
long[] values = longValues[i];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (long value : values) {
defaultSketch.update(value);
}
}
for (int j = 0; j < numFilters; j++) {
if (_filterEvaluators.get(j).evaluate(singleValues, valueTypes, valueArrays, i)) {
- UpdateSketch updateSketch = updateSketches.get(j + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(j + 1);
for (long value : values) {
updateSketch.update(value);
}
@@ -578,17 +578,17 @@ public class DistinctCountThetaSketchAggregationFunction
case FLOAT:
float[][] floatValues = (float[][]) valueArrays[0];
for (int i = 0; i < length; i++) {
- List<UpdateSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
float[] values = floatValues[i];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (float value : values) {
defaultSketch.update(value);
}
}
for (int j = 0; j < numFilters; j++) {
if (_filterEvaluators.get(j).evaluate(singleValues, valueTypes, valueArrays, i)) {
- UpdateSketch updateSketch = updateSketches.get(j + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(j + 1);
for (float value : values) {
updateSketch.update(value);
}
@@ -599,17 +599,17 @@ public class DistinctCountThetaSketchAggregationFunction
case DOUBLE:
double[][] doubleValues = (double[][]) valueArrays[0];
for (int i = 0; i < length; i++) {
- List<UpdateSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
double[] values = doubleValues[i];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (double value : values) {
defaultSketch.update(value);
}
}
for (int j = 0; j < numFilters; j++) {
if (_filterEvaluators.get(j).evaluate(singleValues, valueTypes, valueArrays, i)) {
- UpdateSketch updateSketch = updateSketches.get(j + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(j + 1);
for (double value : values) {
updateSketch.update(value);
}
@@ -620,17 +620,17 @@ public class DistinctCountThetaSketchAggregationFunction
case STRING:
String[][] stringValues = (String[][]) valueArrays[0];
for (int i = 0; i < length; i++) {
- List<UpdateSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
+ List<UpdatableThetaSketch> updateSketches = getUpdateSketches(groupByResultHolder, groupKeyArray[i]);
String[] values = stringValues[i];
if (_includeDefaultSketch) {
- UpdateSketch defaultSketch = updateSketches.get(0);
+ UpdatableThetaSketch defaultSketch = updateSketches.get(0);
for (String value : values) {
defaultSketch.update(value);
}
}
for (int j = 0; j < numFilters; j++) {
if (_filterEvaluators.get(j).evaluate(singleValues, valueTypes, valueArrays, i)) {
- UpdateSketch updateSketch = updateSketches.get(j + 1);
+ UpdatableThetaSketch updateSketch = updateSketches.get(j + 1);
for (String value : values) {
updateSketch.update(value);
}
@@ -645,10 +645,10 @@ public class DistinctCountThetaSketchAggregationFunction
}
} else {
// Serialized sketch
- Sketch[] sketches = deserializeSketches((byte[][]) valueArrays[0], length);
+ ThetaSketch[] sketches = deserializeSketches((byte[][]) valueArrays[0], length);
for (int i = 0; i < length; i++) {
List<ThetaSketchAccumulator> thetaSketchAccumulators = getUnions(groupByResultHolder, groupKeyArray[i]);
- Sketch sketch = sketches[i];
+ ThetaSketch sketch = sketches[i];
if (_includeDefaultSketch) {
thetaSketchAccumulators.get(0).apply(sketch);
}
@@ -787,7 +787,7 @@ public class DistinctCountThetaSketchAggregationFunction
if (_includeDefaultSketch) {
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
- UpdateSketch defaultSketch = getUpdateSketches(groupByResultHolder, groupKey).get(0);
+ UpdatableThetaSketch defaultSketch = getUpdateSketches(groupByResultHolder, groupKey).get(0);
for (int value : intValues[i]) {
defaultSketch.update(value);
}
@@ -799,7 +799,7 @@ public class DistinctCountThetaSketchAggregationFunction
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
for (int groupKey : groupKeysArray[i]) {
- UpdateSketch updateSketch = getUpdateSketches(groupByResultHolder, groupKey).get(i + 1);
+ UpdatableThetaSketch updateSketch = getUpdateSketches(groupByResultHolder, groupKey).get(i + 1);
for (int value : intValues[i]) {
updateSketch.update(value);
}
@@ -813,7 +813,7 @@ public class DistinctCountThetaSketchAggregationFunction
if (_includeDefaultSketch) {
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
- UpdateSketch defaultSketch = getUpdateSketches(groupByResultHolder, groupKey).get(0);
+ UpdatableThetaSketch defaultSketch = getUpdateSketches(groupByResultHolder, groupKey).get(0);
for (long value : longValues[i]) {
defaultSketch.update(value);
}
@@ -825,7 +825,7 @@ public class DistinctCountThetaSketchAggregationFunction
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
for (int groupKey : groupKeysArray[i]) {
- UpdateSketch updateSketch = getUpdateSketches(groupByResultHolder, groupKey).get(i + 1);
+ UpdatableThetaSketch updateSketch = getUpdateSketches(groupByResultHolder, groupKey).get(i + 1);
for (long value : longValues[i]) {
updateSketch.update(value);
}
@@ -839,7 +839,7 @@ public class DistinctCountThetaSketchAggregationFunction
if (_includeDefaultSketch) {
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
- UpdateSketch defaultSketch = getUpdateSketches(groupByResultHolder, groupKey).get(0);
+ UpdatableThetaSketch defaultSketch = getUpdateSketches(groupByResultHolder, groupKey).get(0);
for (float value : floatValues[i]) {
defaultSketch.update(value);
}
@@ -851,7 +851,7 @@ public class DistinctCountThetaSketchAggregationFunction
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
for (int groupKey : groupKeysArray[i]) {
- UpdateSketch updateSketch = getUpdateSketches(groupByResultHolder, groupKey).get(i + 1);
+ UpdatableThetaSketch updateSketch = getUpdateSketches(groupByResultHolder, groupKey).get(i + 1);
for (float value : floatValues[i]) {
updateSketch.update(value);
}
@@ -865,7 +865,7 @@ public class DistinctCountThetaSketchAggregationFunction
if (_includeDefaultSketch) {
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
- UpdateSketch defaultSketch = getUpdateSketches(groupByResultHolder, groupKey).get(0);
+ UpdatableThetaSketch defaultSketch = getUpdateSketches(groupByResultHolder, groupKey).get(0);
for (double value : doubleValues[i]) {
defaultSketch.update(value);
}
@@ -877,7 +877,7 @@ public class DistinctCountThetaSketchAggregationFunction
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
for (int groupKey : groupKeysArray[i]) {
- UpdateSketch updateSketch = getUpdateSketches(groupByResultHolder, groupKey).get(i + 1);
+ UpdatableThetaSketch updateSketch = getUpdateSketches(groupByResultHolder, groupKey).get(i + 1);
for (double value : doubleValues[i]) {
updateSketch.update(value);
}
@@ -891,7 +891,7 @@ public class DistinctCountThetaSketchAggregationFunction
if (_includeDefaultSketch) {
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
- UpdateSketch defaultSketch = getUpdateSketches(groupByResultHolder, groupKey).get(0);
+ UpdatableThetaSketch defaultSketch = getUpdateSketches(groupByResultHolder, groupKey).get(0);
for (String value : stringValues[i]) {
defaultSketch.update(value);
}
@@ -903,7 +903,7 @@ public class DistinctCountThetaSketchAggregationFunction
for (int j = 0; j < length; j++) {
if (filterEvaluator.evaluate(singleValues, valueTypes, valueArrays, j)) {
for (int groupKey : groupKeysArray[i]) {
- UpdateSketch updateSketch = getUpdateSketches(groupByResultHolder, groupKey).get(i + 1);
+ UpdatableThetaSketch updateSketch = getUpdateSketches(groupByResultHolder, groupKey).get(i + 1);
for (String value : stringValues[i]) {
updateSketch.update(value);
}
@@ -919,7 +919,7 @@ public class DistinctCountThetaSketchAggregationFunction
}
} else {
// Serialized sketch
- Sketch[] sketches = deserializeSketches((byte[][]) valueArrays[0], length);
+ ThetaSketch[] sketches = deserializeSketches((byte[][]) valueArrays[0], length);
if (_includeDefaultSketch) {
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
@@ -952,7 +952,7 @@ public class DistinctCountThetaSketchAggregationFunction
return sketches;
}
- if (result.get(0) instanceof Sketch) {
+ if (result.get(0) instanceof ThetaSketch) {
ArrayList<ThetaSketchAccumulator> thetaSketchAccumulators = new ArrayList<>(result.size());
for (Object o : result) {
ThetaSketchAccumulator thetaSketchAccumulator = convertSketchAccumulator(o);
@@ -976,7 +976,7 @@ public class DistinctCountThetaSketchAggregationFunction
return thetaSketchAccumulators;
}
- if (result.get(0) instanceof Sketch) {
+ if (result.get(0) instanceof ThetaSketch) {
ArrayList<ThetaSketchAccumulator> thetaSketchAccumulators = new ArrayList<>(result.size());
for (Object o : result) {
ThetaSketchAccumulator thetaSketchAccumulator = convertSketchAccumulator(o);
@@ -1084,7 +1084,7 @@ public class DistinctCountThetaSketchAggregationFunction
return 0L;
}
int numAccumulators = accumulators.size();
- List<Sketch> mergedSketches = new ArrayList<>(numAccumulators);
+ List<ThetaSketch> mergedSketches = new ArrayList<>(numAccumulators);
for (Object accumulatorObject : accumulators) {
ThetaSketchAccumulator accumulator = convertSketchAccumulator(accumulatorObject);
@@ -1126,8 +1126,8 @@ public class DistinctCountThetaSketchAggregationFunction
// types might still be incompatible at runtime due to type erasure.
// Due to performance overheads of redundant casts, this should be removed at some future point.
protected ThetaSketchAccumulator convertSketchAccumulator(Object result) {
- if (result instanceof Sketch) {
- Sketch sketch = (Sketch) result;
+ if (result instanceof ThetaSketch) {
+ ThetaSketch sketch = (ThetaSketch) result;
ThetaSketchAccumulator accumulator = new ThetaSketchAccumulator(_setOperationBuilder, _accumulatorThreshold);
accumulator.apply(sketch);
return accumulator;
@@ -1300,10 +1300,10 @@ public class DistinctCountThetaSketchAggregationFunction
}
/**
- * Returns the UpdateSketch list from the result holder or creates a new one if it does not exist.
+ * Returns the UpdatableThetaSketch list from the result holder or creates a new one if it does not exist.
*/
- private List<UpdateSketch> getUpdateSketches(AggregationResultHolder aggregationResultHolder) {
- List<UpdateSketch> updateSketches = aggregationResultHolder.getResult();
+ private List<UpdatableThetaSketch> getUpdateSketches(AggregationResultHolder aggregationResultHolder) {
+ List<UpdatableThetaSketch> updateSketches = aggregationResultHolder.getResult();
if (updateSketches == null) {
updateSketches = buildUpdateSketches();
aggregationResultHolder.setValue(updateSketches);
@@ -1312,7 +1312,7 @@ public class DistinctCountThetaSketchAggregationFunction
}
/**
- * Returns the Union list from the result holder or creates a new one if it does not exist.
+ * Returns the ThetaSketchAccumulator list from the result holder or creates a new one if it does not exist.
*/
private List<ThetaSketchAccumulator> getUnions(AggregationResultHolder aggregationResultHolder) {
List<ThetaSketchAccumulator> unions = aggregationResultHolder.getResult();
@@ -1324,10 +1324,10 @@ public class DistinctCountThetaSketchAggregationFunction
}
/**
- * Returns the UpdateSketch list for the given group key or creates a new one if it does not exist.
+ * Returns the UpdatableThetaSketch list for the given group key or creates a new one if it does not exist.
*/
- private List<UpdateSketch> getUpdateSketches(GroupByResultHolder groupByResultHolder, int groupKey) {
- List<UpdateSketch> updateSketches = groupByResultHolder.getResult(groupKey);
+ private List<UpdatableThetaSketch> getUpdateSketches(GroupByResultHolder groupByResultHolder, int groupKey) {
+ List<UpdatableThetaSketch> updateSketches = groupByResultHolder.getResult(groupKey);
if (updateSketches == null) {
updateSketches = buildUpdateSketches();
groupByResultHolder.setValueForKey(groupKey, updateSketches);
@@ -1336,7 +1336,7 @@ public class DistinctCountThetaSketchAggregationFunction
}
/**
- * Returns the Union list for the given group key or creates a new one if it does not exist.
+ * Returns the ThetaSketchAccumulator list for the given group key or creates a new one if it does not exist.
*/
private List<ThetaSketchAccumulator> getUnions(GroupByResultHolder groupByResultHolder, int groupKey) {
List<ThetaSketchAccumulator> unions = groupByResultHolder.getResult(groupKey);
@@ -1348,11 +1348,11 @@ public class DistinctCountThetaSketchAggregationFunction
}
/**
- * Builds the UpdateSketch list.
+ * Builds the UpdatableThetaSketch list.
*/
- private List<UpdateSketch> buildUpdateSketches() {
+ private List<UpdatableThetaSketch> buildUpdateSketches() {
int numSketches = _filterEvaluators.size() + 1;
- List<UpdateSketch> updateSketches = new ArrayList<>(numSketches);
+ List<UpdatableThetaSketch> updateSketches = new ArrayList<>(numSketches);
for (int i = 0; i < numSketches; i++) {
updateSketches.add(_updateSketchBuilder.build());
}
@@ -1360,7 +1360,7 @@ public class DistinctCountThetaSketchAggregationFunction
}
/**
- * Builds the Union list.
+ * Builds the ThetaSketchAccumulator list.
*/
private List<ThetaSketchAccumulator> buildUnions() {
int numUnions = _filterEvaluators.size() + 1;
@@ -1376,10 +1376,10 @@ public class DistinctCountThetaSketchAggregationFunction
/**
* Deserializes the sketches from the bytes.
*/
- private Sketch[] deserializeSketches(byte[][] serializedSketches, int length) {
- Sketch[] sketches = new Sketch[length];
+ private ThetaSketch[] deserializeSketches(byte[][] serializedSketches, int length) {
+ ThetaSketch[] sketches = new ThetaSketch[length];
for (int i = 0; i < length; i++) {
- sketches[i] = Sketch.wrap(Memory.wrap(serializedSketches[i]));
+ sketches[i] = ThetaSketch.wrap(MemorySegment.ofArray(serializedSketches[i]).asReadOnly());
}
return sketches;
}
@@ -1387,14 +1387,14 @@ public class DistinctCountThetaSketchAggregationFunction
/**
* Evaluates the post-aggregation expression.
*/
- protected Sketch evaluatePostAggregationExpression(List<Sketch> sketches) {
+ protected ThetaSketch evaluatePostAggregationExpression(List<ThetaSketch> sketches) {
return evaluatePostAggregationExpression(_postAggregationExpression, sketches);
}
/**
* Evaluates the post-aggregation expression.
*/
- private Sketch evaluatePostAggregationExpression(ExpressionContext expression, List<Sketch> sketches) {
+ private ThetaSketch evaluatePostAggregationExpression(ExpressionContext expression, List<ThetaSketch> sketches) {
if (expression.getType() == ExpressionContext.Type.IDENTIFIER) {
return sketches.get(extractSketchId(expression.getIdentifier()));
}
@@ -1404,19 +1404,19 @@ public class DistinctCountThetaSketchAggregationFunction
List<ExpressionContext> arguments = function.getArguments();
switch (functionName) {
case SET_UNION:
- Union union = _setOperationBuilder.buildUnion();
+ ThetaUnion union = _setOperationBuilder.buildUnion();
for (ExpressionContext argument : arguments) {
union.union(evaluatePostAggregationExpression(argument, sketches));
}
return union.getResult(false, null);
case SET_INTERSECT:
- Intersection intersection = _setOperationBuilder.buildIntersection();
+ ThetaIntersection intersection = _setOperationBuilder.buildIntersection();
for (ExpressionContext argument : arguments) {
intersection.intersect(evaluatePostAggregationExpression(argument, sketches));
}
return intersection.getResult(false, null);
case SET_DIFF:
- AnotB diff = _setOperationBuilder.buildANotB();
+ ThetaAnotB diff = _setOperationBuilder.buildANotB();
diff.setA(evaluatePostAggregationExpression(arguments.get(0), sketches));
diff.notB(evaluatePostAggregationExpression(arguments.get(1), sketches));
return diff.getResult(false, null, false);
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FrequentLongsSketchAggregationFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FrequentLongsSketchAggregationFunction.java
index 1f722822d54..6b5c6759928 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FrequentLongsSketchAggregationFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FrequentLongsSketchAggregationFunction.java
@@ -20,10 +20,10 @@
package org.apache.pinot.core.query.aggregation.function;
import com.google.common.base.Preconditions;
+import java.lang.foreign.MemorySegment;
import java.util.List;
import java.util.Map;
-import org.apache.datasketches.frequencies.LongsSketch;
-import org.apache.datasketches.memory.Memory;
+import org.apache.datasketches.frequencies.FrequentLongsSketch;
import org.apache.pinot.common.CustomObject;
import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.common.utils.DataSchema;
@@ -57,7 +57,7 @@ import org.apache.pinot.spi.data.FieldSpec;
*
* <p>
* If the column type is BYTES, the aggregation function will assume it is a serialized FrequentItems data sketch
- * of type `LongsSketch`and will attempt to deserialize it for merging with other sketch objects.
+ * of type `FrequentLongsSketch` and will attempt to deserialize it for merging with other sketch objects.
* </p>
*
* <p>
@@ -70,7 +70,7 @@ import org.apache.pinot.spi.data.FieldSpec;
* </p>
*/
public class FrequentLongsSketchAggregationFunction
- extends BaseSingleInputAggregationFunction<LongsSketch, Comparable<?>> {
+ extends BaseSingleInputAggregationFunction<FrequentLongsSketch, Comparable<?>> {
protected static final int DEFAULT_MAX_MAP_SIZE = 256;
protected int _maxMapSize;
@@ -104,15 +104,16 @@ public class FrequentLongsSketchAggregationFunction
BlockValSet valueSet = blockValSetMap.get(_expression);
FieldSpec.DataType valueType = valueSet.getValueType();
- LongsSketch sketch = getOrCreateSketch(aggregationResultHolder);
+ FrequentLongsSketch sketch = getOrCreateSketch(aggregationResultHolder);
switch (valueType) {
case BYTES:
// Assuming the column contains serialized data sketch
- LongsSketch[] deserializedSketches = deserializeSketches(blockValSetMap.get(_expression).getBytesValuesSV());
+ FrequentLongsSketch[] deserializedSketches =
+ deserializeSketches(blockValSetMap.get(_expression).getBytesValuesSV());
sketch = getOrCreateSketch(aggregationResultHolder);
- for (LongsSketch colSketch : deserializedSketches) {
+ for (FrequentLongsSketch colSketch : deserializedSketches) {
sketch.merge(colSketch);
}
break;
@@ -136,10 +137,10 @@ public class FrequentLongsSketchAggregationFunction
switch (valueType) {
case BYTES:
// serialized sketch
- LongsSketch[] deserializedSketches =
+ FrequentLongsSketch[] deserializedSketches =
deserializeSketches(blockValSetMap.get(_expression).getBytesValuesSV());
for (int i = 0; i < length; i++) {
- LongsSketch sketch = getOrCreateSketch(groupByResultHolder, groupKeyArray[i]);
+ FrequentLongsSketch sketch = getOrCreateSketch(groupByResultHolder, groupKeyArray[i]);
sketch.merge(deserializedSketches[i]);
}
break;
@@ -147,7 +148,7 @@ public class FrequentLongsSketchAggregationFunction
case LONG:
long[] values = valueSet.getLongValuesSV();
for (int i = 0; i < length; i++) {
- LongsSketch sketch = getOrCreateSketch(groupByResultHolder, groupKeyArray[i]);
+ FrequentLongsSketch sketch = getOrCreateSketch(groupByResultHolder, groupKeyArray[i]);
sketch.update(values[i]);
}
break;
@@ -165,11 +166,11 @@ public class FrequentLongsSketchAggregationFunction
switch (valueType) {
case BYTES:
// serialized sketch
- LongsSketch[] deserializedSketches =
+ FrequentLongsSketch[] deserializedSketches =
deserializeSketches(blockValSetMap.get(_expression).getBytesValuesSV());
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
- LongsSketch sketch = getOrCreateSketch(groupByResultHolder, groupKey);
+ FrequentLongsSketch sketch = getOrCreateSketch(groupByResultHolder, groupKey);
sketch.merge(deserializedSketches[i]);
}
}
@@ -179,7 +180,7 @@ public class FrequentLongsSketchAggregationFunction
long[] values = valueSet.getLongValuesSV();
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
- LongsSketch sketch = getOrCreateSketch(groupByResultHolder, groupKey);
+ FrequentLongsSketch sketch = getOrCreateSketch(groupByResultHolder, groupKey);
sketch.update(values[i]);
}
}
@@ -192,10 +193,10 @@ public class FrequentLongsSketchAggregationFunction
/**
* Extracts the sketch from the result holder or creates a new one if it does not exist.
*/
- protected LongsSketch getOrCreateSketch(AggregationResultHolder aggregationResultHolder) {
- LongsSketch sketch = aggregationResultHolder.getResult();
+ protected FrequentLongsSketch getOrCreateSketch(AggregationResultHolder aggregationResultHolder) {
+ FrequentLongsSketch sketch = aggregationResultHolder.getResult();
if (sketch == null) {
- sketch = new LongsSketch(_maxMapSize);
+ sketch = new FrequentLongsSketch(_maxMapSize);
aggregationResultHolder.setValue(sketch);
}
return sketch;
@@ -205,10 +206,10 @@ public class FrequentLongsSketchAggregationFunction
* Extracts the sketch from the group by result holder for key
* or creates a new one if it does not exist.
*/
- protected LongsSketch getOrCreateSketch(GroupByResultHolder groupByResultHolder, int groupKey) {
- LongsSketch sketch = groupByResultHolder.getResult(groupKey);
+ protected FrequentLongsSketch getOrCreateSketch(GroupByResultHolder groupByResultHolder, int groupKey) {
+ FrequentLongsSketch sketch = groupByResultHolder.getResult(groupKey);
if (sketch == null) {
- sketch = new LongsSketch(_maxMapSize);
+ sketch = new FrequentLongsSketch(_maxMapSize);
groupByResultHolder.setValueForKey(groupKey, sketch);
}
return sketch;
@@ -217,27 +218,27 @@ public class FrequentLongsSketchAggregationFunction
/**
* Deserializes the sketches from the bytes.
*/
- protected LongsSketch[] deserializeSketches(byte[][] serializedSketches) {
- LongsSketch[] sketches = new LongsSketch[serializedSketches.length];
+ protected FrequentLongsSketch[] deserializeSketches(byte[][] serializedSketches) {
+ FrequentLongsSketch[] sketches = new FrequentLongsSketch[serializedSketches.length];
for (int i = 0; i < serializedSketches.length; i++) {
- sketches[i] = LongsSketch.getInstance(Memory.wrap(serializedSketches[i]));
+ sketches[i] = FrequentLongsSketch.getInstance(MemorySegment.ofArray(serializedSketches[i]));
}
return sketches;
}
@Override
- public LongsSketch extractAggregationResult(AggregationResultHolder aggregationResultHolder) {
+ public FrequentLongsSketch extractAggregationResult(AggregationResultHolder aggregationResultHolder) {
return aggregationResultHolder.getResult();
}
@Override
- public LongsSketch extractGroupByResult(GroupByResultHolder groupByResultHolder, int groupKey) {
+ public FrequentLongsSketch extractGroupByResult(GroupByResultHolder groupByResultHolder, int groupKey) {
return groupByResultHolder.getResult(groupKey);
}
@Override
- public LongsSketch merge(LongsSketch sketch1, LongsSketch sketch2) {
- LongsSketch union = new LongsSketch(_maxMapSize);
+ public FrequentLongsSketch merge(FrequentLongsSketch sketch1, FrequentLongsSketch sketch2) {
+ FrequentLongsSketch union = new FrequentLongsSketch(_maxMapSize);
if (sketch1 != null) {
union.merge(sketch1);
}
@@ -253,13 +254,13 @@ public class FrequentLongsSketchAggregationFunction
}
@Override
- public SerializedIntermediateResult serializeIntermediateResult(LongsSketch longsSketch) {
+ public SerializedIntermediateResult serializeIntermediateResult(FrequentLongsSketch longsSketch) {
return new SerializedIntermediateResult(ObjectSerDeUtils.ObjectType.FrequentLongsSketch.getValue(),
ObjectSerDeUtils.FREQUENT_LONGS_SKETCH_SER_DE.serialize(longsSketch));
}
@Override
- public LongsSketch deserializeIntermediateResult(CustomObject customObject) {
+ public FrequentLongsSketch deserializeIntermediateResult(CustomObject customObject) {
return ObjectSerDeUtils.FREQUENT_LONGS_SKETCH_SER_DE.deserialize(customObject.getBuffer());
}
@@ -275,7 +276,7 @@ public class FrequentLongsSketchAggregationFunction
}
@Override
- public Comparable<?> extractFinalResult(LongsSketch sketch) {
+ public Comparable<?> extractFinalResult(FrequentLongsSketch sketch) {
return new SerializedFrequentLongsSketch(sketch);
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FrequentStringsSketchAggregationFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FrequentStringsSketchAggregationFunction.java
index 41fe3050f02..006e9544604 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FrequentStringsSketchAggregationFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FrequentStringsSketchAggregationFunction.java
@@ -20,11 +20,11 @@
package org.apache.pinot.core.query.aggregation.function;
import com.google.common.base.Preconditions;
+import java.lang.foreign.MemorySegment;
import java.util.List;
import java.util.Map;
import org.apache.datasketches.common.ArrayOfStringsSerDe;
-import org.apache.datasketches.frequencies.ItemsSketch;
-import org.apache.datasketches.memory.Memory;
+import org.apache.datasketches.frequencies.FrequentItemsSketch;
import org.apache.pinot.common.CustomObject;
import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.common.utils.DataSchema;
@@ -71,7 +71,7 @@ import org.apache.pinot.spi.data.FieldSpec;
* </p>
*/
public class FrequentStringsSketchAggregationFunction
- extends BaseSingleInputAggregationFunction<ItemsSketch<String>, Comparable<?>> {
+ extends BaseSingleInputAggregationFunction<FrequentItemsSketch<String>, Comparable<?>> {
protected static final int DEFAULT_MAX_MAP_SIZE = 256;
protected int _maxMapSize;
@@ -105,15 +105,15 @@ public class FrequentStringsSketchAggregationFunction
BlockValSet valueSet = blockValSetMap.get(_expression);
FieldSpec.DataType valueType = valueSet.getValueType();
- ItemsSketch<String> sketch = getOrCreateSketch(aggregationResultHolder);
+ FrequentItemsSketch<String> sketch = getOrCreateSketch(aggregationResultHolder);
if (valueType == FieldSpec.DataType.BYTES) {
// Assuming the column contains serialized data sketch
- ItemsSketch<String>[] deserializedSketches =
+ FrequentItemsSketch<String>[] deserializedSketches =
deserializeSketches(blockValSetMap.get(_expression).getBytesValuesSV());
sketch = getOrCreateSketch(aggregationResultHolder);
- for (ItemsSketch<String> colSketch : deserializedSketches) {
+ for (FrequentItemsSketch<String> colSketch : deserializedSketches) {
sketch.merge(colSketch);
}
} else {
@@ -131,16 +131,16 @@ public class FrequentStringsSketchAggregationFunction
if (valueType == FieldSpec.DataType.BYTES) {
// serialized sketch
- ItemsSketch<String>[] deserializedSketches =
+ FrequentItemsSketch<String>[] deserializedSketches =
deserializeSketches(blockValSetMap.get(_expression).getBytesValuesSV());
for (int i = 0; i < length; i++) {
- ItemsSketch<String> sketch = getOrCreateSketch(groupByResultHolder, groupKeyArray[i]);
+ FrequentItemsSketch<String> sketch = getOrCreateSketch(groupByResultHolder, groupKeyArray[i]);
sketch.merge(deserializedSketches[i]);
}
} else {
String[] values = valueSet.getStringValuesSV();
for (int i = 0; i < length; i++) {
- ItemsSketch<String> sketch = getOrCreateSketch(groupByResultHolder, groupKeyArray[i]);
+ FrequentItemsSketch<String> sketch = getOrCreateSketch(groupByResultHolder, groupKeyArray[i]);
sketch.update(values[i]);
}
}
@@ -154,11 +154,11 @@ public class FrequentStringsSketchAggregationFunction
if (valueType == FieldSpec.DataType.BYTES) {
// serialized sketch
- ItemsSketch<String>[] deserializedSketches =
+ FrequentItemsSketch<String>[] deserializedSketches =
deserializeSketches(blockValSetMap.get(_expression).getBytesValuesSV());
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
- ItemsSketch<String> sketch = getOrCreateSketch(groupByResultHolder, groupKey);
+ FrequentItemsSketch<String> sketch = getOrCreateSketch(groupByResultHolder, groupKey);
sketch.merge(deserializedSketches[i]);
}
}
@@ -166,7 +166,7 @@ public class FrequentStringsSketchAggregationFunction
String[] values = valueSet.getStringValuesSV();
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
- ItemsSketch<String> sketch = getOrCreateSketch(groupByResultHolder, groupKey);
+ FrequentItemsSketch<String> sketch = getOrCreateSketch(groupByResultHolder, groupKey);
sketch.update(values[i]);
}
}
@@ -176,10 +176,10 @@ public class FrequentStringsSketchAggregationFunction
/**
* Extracts the sketch from the result holder or creates a new one if it does not exist.
*/
- protected ItemsSketch<String> getOrCreateSketch(AggregationResultHolder aggregationResultHolder) {
- ItemsSketch<String> sketch = aggregationResultHolder.getResult();
+ protected FrequentItemsSketch<String> getOrCreateSketch(AggregationResultHolder aggregationResultHolder) {
+ FrequentItemsSketch<String> sketch = aggregationResultHolder.getResult();
if (sketch == null) {
- sketch = new ItemsSketch<>(_maxMapSize);
+ sketch = new FrequentItemsSketch<>(_maxMapSize);
aggregationResultHolder.setValue(sketch);
}
return sketch;
@@ -189,10 +189,10 @@ public class FrequentStringsSketchAggregationFunction
* Extracts the sketch from the group by result holder for key
* or creates a new one if it does not exist.
*/
- protected ItemsSketch<String> getOrCreateSketch(GroupByResultHolder groupByResultHolder, int groupKey) {
- ItemsSketch<String> sketch = groupByResultHolder.getResult(groupKey);
+ protected FrequentItemsSketch<String> getOrCreateSketch(GroupByResultHolder groupByResultHolder, int groupKey) {
+ FrequentItemsSketch<String> sketch = groupByResultHolder.getResult(groupKey);
if (sketch == null) {
- sketch = new ItemsSketch<>(_maxMapSize);
+ sketch = new FrequentItemsSketch<>(_maxMapSize);
groupByResultHolder.setValueForKey(groupKey, sketch);
}
return sketch;
@@ -201,27 +201,28 @@ public class FrequentStringsSketchAggregationFunction
/**
* Deserializes the sketches from the bytes.
*/
- protected ItemsSketch<String>[] deserializeSketches(byte[][] serializedSketches) {
- ItemsSketch<String>[] sketches = new ItemsSketch[serializedSketches.length];
+ protected FrequentItemsSketch<String>[] deserializeSketches(byte[][] serializedSketches) {
+ FrequentItemsSketch<String>[] sketches = new FrequentItemsSketch[serializedSketches.length];
for (int i = 0; i < serializedSketches.length; i++) {
- sketches[i] = ItemsSketch.getInstance(Memory.wrap(serializedSketches[i]), new ArrayOfStringsSerDe());
+ sketches[i] =
+ FrequentItemsSketch.getInstance(MemorySegment.ofArray(serializedSketches[i]), new ArrayOfStringsSerDe());
}
return sketches;
}
@Override
- public ItemsSketch<String> extractAggregationResult(AggregationResultHolder aggregationResultHolder) {
+ public FrequentItemsSketch<String> extractAggregationResult(AggregationResultHolder aggregationResultHolder) {
return aggregationResultHolder.getResult();
}
@Override
- public ItemsSketch<String> extractGroupByResult(GroupByResultHolder groupByResultHolder, int groupKey) {
+ public FrequentItemsSketch<String> extractGroupByResult(GroupByResultHolder groupByResultHolder, int groupKey) {
return groupByResultHolder.getResult(groupKey);
}
@Override
- public ItemsSketch<String> merge(ItemsSketch<String> sketch1, ItemsSketch<String> sketch2) {
- ItemsSketch<String> union = new ItemsSketch<>(_maxMapSize);
+ public FrequentItemsSketch<String> merge(FrequentItemsSketch<String> sketch1, FrequentItemsSketch<String> sketch2) {
+ FrequentItemsSketch<String> union = new FrequentItemsSketch<>(_maxMapSize);
if (sketch1 != null) {
union.merge(sketch1);
}
@@ -237,13 +238,13 @@ public class FrequentStringsSketchAggregationFunction
}
@Override
- public SerializedIntermediateResult serializeIntermediateResult(ItemsSketch<String> stringItemsSketch) {
+ public SerializedIntermediateResult serializeIntermediateResult(FrequentItemsSketch<String> stringItemsSketch) {
return new SerializedIntermediateResult(ObjectSerDeUtils.ObjectType.FrequentStringsSketch.getValue(),
ObjectSerDeUtils.FREQUENT_STRINGS_SKETCH_SER_DE.serialize(stringItemsSketch));
}
@Override
- public ItemsSketch<String> deserializeIntermediateResult(CustomObject customObject) {
+ public FrequentItemsSketch<String> deserializeIntermediateResult(CustomObject customObject) {
return ObjectSerDeUtils.FREQUENT_STRINGS_SKETCH_SER_DE.deserialize(customObject.getBuffer());
}
@@ -259,7 +260,7 @@ public class FrequentStringsSketchAggregationFunction
}
@Override
- public Comparable<?> extractFinalResult(ItemsSketch<String> sketch) {
+ public Comparable<?> extractFinalResult(FrequentItemsSketch<String> sketch) {
return new SerializedFrequentStringsSketch(sketch);
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/IntegerTupleSketchAggregationFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/IntegerTupleSketchAggregationFunction.java
index 679933c5ed1..5e3e31f99ac 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/IntegerTupleSketchAggregationFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/IntegerTupleSketchAggregationFunction.java
@@ -19,14 +19,13 @@
package org.apache.pinot.core.query.aggregation.function;
import com.google.common.base.Preconditions;
+import java.lang.foreign.MemorySegment;
import java.util.Base64;
import java.util.List;
import java.util.Map;
import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
-import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.tuple.Sketch;
-import org.apache.datasketches.tuple.Sketches;
+import org.apache.datasketches.tuple.TupleSketch;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.datasketches.tuple.aninteger.IntegerSummaryDeserializer;
import org.apache.datasketches.tuple.aninteger.IntegerSummarySetOperations;
@@ -165,8 +164,8 @@ public class IntegerTupleSketchAggregationFunction
byte[][] bytesValues = blockValSet.getBytesValuesSV();
try {
TupleIntSketchAccumulator tupleIntSketchAccumulator = getAccumulator(aggregationResultHolder);
- Sketch<IntegerSummary>[] sketches = deserializeSketches(bytesValues, length);
- for (Sketch<IntegerSummary> sketch : sketches) {
+ TupleSketch<IntegerSummary>[] sketches = deserializeSketches(bytesValues, length);
+ for (TupleSketch<IntegerSummary> sketch : sketches) {
tupleIntSketchAccumulator.apply(sketch);
}
} catch (Exception e) {
@@ -189,10 +188,10 @@ public class IntegerTupleSketchAggregationFunction
if (storedType == FieldSpec.DataType.BYTES) {
byte[][] bytesValues = blockValSet.getBytesValuesSV();
try {
- Sketch<IntegerSummary>[] sketches = deserializeSketches(bytesValues, length);
+ TupleSketch<IntegerSummary>[] sketches = deserializeSketches(bytesValues, length);
for (int i = 0; i < length; i++) {
TupleIntSketchAccumulator tupleIntSketchAccumulator = getAccumulator(groupByResultHolder, groupKeyArray[i]);
- Sketch<IntegerSummary> sketch = sketches[i];
+ TupleSketch<IntegerSummary> sketch = sketches[i];
tupleIntSketchAccumulator.apply(sketch);
}
} catch (Exception e) {
@@ -216,7 +215,7 @@ public class IntegerTupleSketchAggregationFunction
if (singleValue && storedType == FieldSpec.DataType.BYTES) {
byte[][] bytesValues = blockValSetMap.get(_expression).getBytesValuesSV();
try {
- Sketch<IntegerSummary>[] sketches = deserializeSketches(bytesValues, length);
+ TupleSketch<IntegerSummary>[] sketches = deserializeSketches(bytesValues, length);
for (int i = 0; i < length; i++) {
for (int groupKey : groupKeysArray[i]) {
getAccumulator(groupByResultHolder, groupKey).apply(sketches[i]);
@@ -342,10 +341,11 @@ public class IntegerTupleSketchAggregationFunction
* Deserializes the sketches from the bytes.
*/
@SuppressWarnings({"unchecked"})
- private Sketch<IntegerSummary>[] deserializeSketches(byte[][] serializedSketches, int length) {
- Sketch<IntegerSummary>[] sketches = new Sketch[length];
+ private TupleSketch<IntegerSummary>[] deserializeSketches(byte[][] serializedSketches, int length) {
+ TupleSketch<IntegerSummary>[] sketches = new TupleSketch[length];
for (int i = 0; i < length; i++) {
- sketches[i] = Sketches.heapifySketch(Memory.wrap(serializedSketches[i]), new IntegerSummaryDeserializer());
+ sketches[i] =
+ TupleSketch.heapifySketch(MemorySegment.ofArray(serializedSketches[i]), new IntegerSummaryDeserializer());
}
return sketches;
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileKLLAggregationFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileKLLAggregationFunction.java
index 21498595ad1..ff5ab652dd9 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileKLLAggregationFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/PercentileKLLAggregationFunction.java
@@ -19,10 +19,10 @@
package org.apache.pinot.core.query.aggregation.function;
import com.google.common.base.Preconditions;
+import java.lang.foreign.MemorySegment;
import java.util.List;
import java.util.Map;
import org.apache.datasketches.kll.KllDoublesSketch;
-import org.apache.datasketches.memory.Memory;
import org.apache.pinot.common.CustomObject;
import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.common.utils.DataSchema.ColumnDataType;
@@ -283,7 +283,7 @@ public class PercentileKLLAggregationFunction
protected KllDoublesSketch[] deserializeSketches(byte[][] serializedSketches) {
KllDoublesSketch[] sketches = new KllDoublesSketch[serializedSketches.length];
for (int i = 0; i < serializedSketches.length; i++) {
- sketches[i] = KllDoublesSketch.wrap(Memory.wrap(serializedSketches[i]));
+ sketches[i] = KllDoublesSketch.wrap(MemorySegment.ofArray(serializedSketches[i]).asReadOnly());
}
return sketches;
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/SumValuesIntegerTupleSketchAggregationFunction.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/SumValuesIntegerTupleSketchAggregationFunction.java
index fa4ac2d68db..d55992b09a4 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/SumValuesIntegerTupleSketchAggregationFunction.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/SumValuesIntegerTupleSketchAggregationFunction.java
@@ -19,7 +19,7 @@
package org.apache.pinot.core.query.aggregation.function;
import java.util.List;
-import org.apache.datasketches.tuple.Sketch;
+import org.apache.datasketches.tuple.TupleSketch;
import org.apache.datasketches.tuple.TupleSketchIterator;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.pinot.common.request.context.ExpressionContext;
@@ -51,7 +51,7 @@ public class SumValuesIntegerTupleSketchAggregationFunction extends IntegerTuple
accumulator.setNominalEntries(_nominalEntries);
accumulator.setSetOperations(_setOps);
accumulator.setThreshold(_accumulatorThreshold);
- Sketch<IntegerSummary> result = accumulator.getResult();
+ TupleSketch<IntegerSummary> result = accumulator.getResult();
TupleSketchIterator<IntegerSummary> summaries = result.iterator();
while (summaries.next()) {
retainedTotal += summaries.getSummary().getValue();
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/FunnelCountAggregationFunctionFactory.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/FunnelCountAggregationFunctionFactory.java
index 91e9bfca51a..48c7d1b99c0 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/FunnelCountAggregationFunctionFactory.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/FunnelCountAggregationFunctionFactory.java
@@ -27,8 +27,8 @@ import java.util.Set;
import java.util.function.Supplier;
import java.util.stream.Collectors;
import org.apache.commons.lang3.StringUtils;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.UpdateSketch;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
import org.apache.datasketches.thetacommon.ThetaUtil;
import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
@@ -130,7 +130,7 @@ public class FunnelCountAggregationFunctionFactory implements Supplier<Aggregati
}
}
- AggregationStrategy<UpdateSketch[]> thetaSketchAggregationStrategy() {
+ AggregationStrategy<UpdatableThetaSketch[]> thetaSketchAggregationStrategy() {
return new ThetaSketchAggregationStrategy(_stepExpressions, _correlateByExpressions, _nominalEntries);
}
@@ -138,7 +138,7 @@ public class FunnelCountAggregationFunctionFactory implements Supplier<Aggregati
return new BitmapAggregationStrategy(_stepExpressions, _correlateByExpressions);
}
- MergeStrategy<List<Sketch>> thetaSketchMergeStrategy() {
+ MergeStrategy<List<ThetaSketch>> thetaSketchMergeStrategy() {
return new ThetaSketchMergeStrategy(_numSteps, _nominalEntries);
}
@@ -154,7 +154,7 @@ public class FunnelCountAggregationFunctionFactory implements Supplier<Aggregati
return new PartitionedMergeStrategy(_numSteps);
}
- ResultExtractionStrategy<UpdateSketch[], List<Sketch>> thetaSketchResultExtractionStrategy() {
+ ResultExtractionStrategy<UpdatableThetaSketch[], List<ThetaSketch>> thetaSketchResultExtractionStrategy() {
return new ThetaSketchResultExtractionStrategy(_numSteps);
}
@@ -179,8 +179,8 @@ public class FunnelCountAggregationFunctionFactory implements Supplier<Aggregati
};
}
- ResultExtractionStrategy<UpdateSketch[], List<Long>> thetaSketchPartitionedResultExtractionStrategy() {
- final MergeStrategy<List<Sketch>> thetaSketchMergeStrategy = thetaSketchMergeStrategy();
+ ResultExtractionStrategy<UpdatableThetaSketch[], List<Long>> thetaSketchPartitionedResultExtractionStrategy() {
+ final MergeStrategy<List<ThetaSketch>> thetaSketchMergeStrategy = thetaSketchMergeStrategy();
return sketches -> {
if (sketches == null) {
return Collections.nCopies(_numSteps, 0L);
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchAggregationStrategy.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchAggregationStrategy.java
index da16056705b..e3b92b9c453 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchAggregationStrategy.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchAggregationStrategy.java
@@ -19,8 +19,8 @@
package org.apache.pinot.core.query.aggregation.function.funnel;
import java.util.List;
-import org.apache.datasketches.theta.UpdateSketch;
-import org.apache.datasketches.theta.UpdateSketchBuilder;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketchBuilder;
import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.segment.spi.index.reader.Dictionary;
@@ -28,18 +28,18 @@ import org.apache.pinot.segment.spi.index.reader.Dictionary;
/**
* Aggregation strategy leveraging theta sketch algebra (unions/intersections).
*/
-class ThetaSketchAggregationStrategy extends AggregationStrategy<UpdateSketch[]> {
- final UpdateSketchBuilder _updateSketchBuilder;
+class ThetaSketchAggregationStrategy extends AggregationStrategy<UpdatableThetaSketch[]> {
+ final UpdatableThetaSketchBuilder _updateSketchBuilder;
public ThetaSketchAggregationStrategy(List<ExpressionContext> stepExpressions,
List<ExpressionContext> correlateByExpressions, int nominalEntries) {
super(stepExpressions, correlateByExpressions);
- _updateSketchBuilder = new UpdateSketchBuilder().setNominalEntries(nominalEntries);
+ _updateSketchBuilder = new UpdatableThetaSketchBuilder().setNominalEntries(nominalEntries);
}
@Override
- public UpdateSketch[] createAggregationResult(Dictionary dictionary) {
- final UpdateSketch[] stepsSketches = new UpdateSketch[_numSteps];
+ public UpdatableThetaSketch[] createAggregationResult(Dictionary dictionary) {
+ final UpdatableThetaSketch[] stepsSketches = new UpdatableThetaSketch[_numSteps];
for (int n = 0; n < _numSteps; n++) {
stepsSketches[n] = _updateSketchBuilder.build();
}
@@ -47,8 +47,8 @@ class ThetaSketchAggregationStrategy extends AggregationStrategy<UpdateSketch[]>
}
@Override
- public UpdateSketch[] createAggregationResultMultiKey(Dictionary[] dictionaries) {
- final UpdateSketch[] stepsSketches = new UpdateSketch[_numSteps];
+ public UpdatableThetaSketch[] createAggregationResultMultiKey(Dictionary[] dictionaries) {
+ final UpdatableThetaSketch[] stepsSketches = new UpdatableThetaSketch[_numSteps];
for (int n = 0; n < _numSteps; n++) {
stepsSketches[n] = _updateSketchBuilder.build();
}
@@ -56,8 +56,8 @@ class ThetaSketchAggregationStrategy extends AggregationStrategy<UpdateSketch[]>
}
@Override
- void add(Dictionary dictionary, UpdateSketch[] stepsSketches, int step, int correlationId) {
- final UpdateSketch sketch = stepsSketches[step];
+ void add(Dictionary dictionary, UpdatableThetaSketch[] stepsSketches, int step, int correlationId) {
+ final UpdatableThetaSketch sketch = stepsSketches[step];
switch (dictionary.getValueType()) {
case INT:
sketch.update(dictionary.getIntValue(correlationId));
@@ -82,7 +82,8 @@ class ThetaSketchAggregationStrategy extends AggregationStrategy<UpdateSketch[]>
}
@Override
- void addMultiKey(UpdateSketch[] stepsSketches, int step, Dictionary[] dictionaries, int[] correlationDictIds) {
+ void addMultiKey(UpdatableThetaSketch[] stepsSketches, int step, Dictionary[] dictionaries,
+ int[] correlationDictIds) {
stepsSketches[step].update(DictIdsWrapper.toCompositeString(dictionaries, correlationDictIds));
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchMergeStrategy.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchMergeStrategy.java
index 80378d7078d..1ec1017b859 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchMergeStrategy.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchMergeStrategy.java
@@ -21,23 +21,23 @@ package org.apache.pinot.core.query.aggregation.function.funnel;
import it.unimi.dsi.fastutil.longs.LongArrayList;
import java.util.ArrayList;
import java.util.List;
-import org.apache.datasketches.theta.Intersection;
-import org.apache.datasketches.theta.SetOperationBuilder;
-import org.apache.datasketches.theta.Sketch;
+import org.apache.datasketches.theta.ThetaIntersection;
+import org.apache.datasketches.theta.ThetaSetOperationBuilder;
+import org.apache.datasketches.theta.ThetaSketch;
-class ThetaSketchMergeStrategy implements MergeStrategy<List<Sketch>> {
+class ThetaSketchMergeStrategy implements MergeStrategy<List<ThetaSketch>> {
protected final int _numSteps;
- final SetOperationBuilder _setOperationBuilder;
+ final ThetaSetOperationBuilder _setOperationBuilder;
ThetaSketchMergeStrategy(int numSteps, int nominalEntries) {
_numSteps = numSteps;
- _setOperationBuilder = new SetOperationBuilder().setNominalEntries(nominalEntries);
+ _setOperationBuilder = new ThetaSetOperationBuilder().setNominalEntries(nominalEntries);
}
@Override
- public List<Sketch> merge(List<Sketch> sketches1, List<Sketch> sketches2) {
- final List<Sketch> mergedSketches = new ArrayList<>(_numSteps);
+ public List<ThetaSketch> merge(List<ThetaSketch> sketches1, List<ThetaSketch> sketches2) {
+ final List<ThetaSketch> mergedSketches = new ArrayList<>(_numSteps);
for (int i = 0; i < _numSteps; i++) {
// NOTE: Compact the sketch in unsorted, on-heap fashion for performance concern.
// See https://datasketches.apache.org/docs/Theta/ThetaSize.html for more details.
@@ -47,13 +47,13 @@ class ThetaSketchMergeStrategy implements MergeStrategy<List<Sketch>> {
}
@Override
- public LongArrayList extractFinalResult(List<Sketch> sketches) {
+ public LongArrayList extractFinalResult(List<ThetaSketch> sketches) {
long[] result = new long[_numSteps];
- Sketch sketch = sketches.get(0);
+ ThetaSketch sketch = sketches.get(0);
result[0] = Math.round(sketch.getEstimate());
for (int i = 1; i < _numSteps; i++) {
- Intersection intersection = _setOperationBuilder.buildIntersection();
+ ThetaIntersection intersection = _setOperationBuilder.buildIntersection();
sketch = intersection.intersect(sketch, sketches.get(i));
result[i] = Math.round(sketch.getEstimate());
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchResultExtractionStrategy.java b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchResultExtractionStrategy.java
index 2c794271bf2..7ece0ec4365 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchResultExtractionStrategy.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/funnel/ThetaSketchResultExtractionStrategy.java
@@ -21,13 +21,14 @@ package org.apache.pinot.core.query.aggregation.function.funnel;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.UpdateSketch;
-import org.apache.datasketches.theta.UpdateSketchBuilder;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketchBuilder;
-class ThetaSketchResultExtractionStrategy implements ResultExtractionStrategy<UpdateSketch[], List<Sketch>> {
- private static final Sketch EMPTY_SKETCH = new UpdateSketchBuilder().build().compact();
+class ThetaSketchResultExtractionStrategy
+ implements ResultExtractionStrategy<UpdatableThetaSketch[], List<ThetaSketch>> {
+ private static final ThetaSketch EMPTY_SKETCH = new UpdatableThetaSketchBuilder().build().compact();
protected final int _numSteps;
@@ -36,7 +37,7 @@ class ThetaSketchResultExtractionStrategy implements ResultExtractionStrategy<Up
}
@Override
- public List<Sketch> extractIntermediateResult(UpdateSketch[] stepsSketches) {
+ public List<ThetaSketch> extractIntermediateResult(UpdatableThetaSketch[] stepsSketches) {
if (stepsSketches == null) {
return Collections.nCopies(_numSteps, EMPTY_SKETCH);
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/processing/aggregator/DistinctCountThetaSketchAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/processing/aggregator/DistinctCountThetaSketchAggregator.java
index 3d00e602f03..4cc4fdf5011 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/processing/aggregator/DistinctCountThetaSketchAggregator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/processing/aggregator/DistinctCountThetaSketchAggregator.java
@@ -19,9 +19,9 @@
package org.apache.pinot.core.segment.processing.aggregator;
import java.util.Map;
-import org.apache.datasketches.theta.SetOperationBuilder;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.Union;
+import org.apache.datasketches.theta.ThetaSetOperationBuilder;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.ThetaUnion;
import org.apache.pinot.core.common.ObjectSerDeUtils;
import org.apache.pinot.segment.spi.Constants;
import org.apache.pinot.spi.utils.CommonConstants;
@@ -34,7 +34,7 @@ public class DistinctCountThetaSketchAggregator implements ValueAggregator {
@Override
public Object aggregate(Object value1, Object value2, Map<String, String> functionParameters) {
- SetOperationBuilder unionBuilder = Union.builder();
+ ThetaSetOperationBuilder unionBuilder = ThetaUnion.builder();
String samplingProbabilityParam = functionParameters.get(Constants.THETA_TUPLE_SKETCH_SAMPLING_PROBABILITY);
String nominalEntriesParam = functionParameters.get(Constants.THETA_TUPLE_SKETCH_NOMINAL_ENTRIES);
@@ -53,10 +53,10 @@ public class DistinctCountThetaSketchAggregator implements ValueAggregator {
unionBuilder.setP(Float.parseFloat(samplingProbabilityParam));
}
- Union union = unionBuilder.buildUnion();
- Sketch first = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize((byte[]) value1);
- Sketch second = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize((byte[]) value2);
- Sketch result = union.union(first, second);
+ ThetaUnion union = unionBuilder.buildUnion();
+ ThetaSketch first = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize((byte[]) value1);
+ ThetaSketch second = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize((byte[]) value2);
+ ThetaSketch result = union.union(first, second);
return ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(result);
}
}
diff --git a/pinot-core/src/main/java/org/apache/pinot/core/segment/processing/aggregator/IntegerTupleSketchAggregator.java b/pinot-core/src/main/java/org/apache/pinot/core/segment/processing/aggregator/IntegerTupleSketchAggregator.java
index 9c1588c74ff..558f59c1fe5 100644
--- a/pinot-core/src/main/java/org/apache/pinot/core/segment/processing/aggregator/IntegerTupleSketchAggregator.java
+++ b/pinot-core/src/main/java/org/apache/pinot/core/segment/processing/aggregator/IntegerTupleSketchAggregator.java
@@ -19,8 +19,8 @@
package org.apache.pinot.core.segment.processing.aggregator;
import java.util.Map;
-import org.apache.datasketches.tuple.Sketch;
-import org.apache.datasketches.tuple.Union;
+import org.apache.datasketches.tuple.TupleSketch;
+import org.apache.datasketches.tuple.TupleUnion;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.datasketches.tuple.aninteger.IntegerSummarySetOperations;
import org.apache.pinot.core.common.ObjectSerDeUtils;
@@ -39,22 +39,22 @@ public class IntegerTupleSketchAggregator implements ValueAggregator {
public Object aggregate(Object value1, Object value2, Map<String, String> functionParameters) {
String nominalEntriesParam = functionParameters.get(Constants.THETA_TUPLE_SKETCH_NOMINAL_ENTRIES);
- Union<IntegerSummary> integerUnion;
+ TupleUnion<IntegerSummary> integerUnion;
IntegerSummarySetOperations setOperations = new IntegerSummarySetOperations(_mode, _mode);
// Check if nominal entries is set
if (nominalEntriesParam != null) {
- integerUnion = new Union<>(Integer.parseInt(nominalEntriesParam), setOperations);
+ integerUnion = new TupleUnion<>(Integer.parseInt(nominalEntriesParam), setOperations);
} else {
// If the functionParameters don't have an explicit nominal entries value set,
// use the default value for nominal entries
int sketchNominalEntries = (int) Math.pow(2, CommonConstants.Helix.DEFAULT_TUPLE_SKETCH_LGK);
- integerUnion = new Union<>(sketchNominalEntries, setOperations);
+ integerUnion = new TupleUnion<>(sketchNominalEntries, setOperations);
}
- Sketch<IntegerSummary> first = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize((byte[]) value1);
- Sketch<IntegerSummary> second = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize((byte[]) value2);
- Sketch<IntegerSummary> result = integerUnion.union(first, second);
+ TupleSketch<IntegerSummary> first = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize((byte[]) value1);
+ TupleSketch<IntegerSummary> second = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize((byte[]) value2);
+ TupleSketch<IntegerSummary> result = integerUnion.union(first, second);
return ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.serialize(result);
}
}
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/common/DataSketchesSerialCompatTest.java b/pinot-core/src/test/java/org/apache/pinot/core/common/DataSketchesSerialCompatTest.java
new file mode 100644
index 00000000000..c483dfca571
--- /dev/null
+++ b/pinot-core/src/test/java/org/apache/pinot/core/common/DataSketchesSerialCompatTest.java
@@ -0,0 +1,540 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.common;
+
+import java.nio.ByteBuffer;
+import java.util.Base64;
+import org.apache.datasketches.cpc.CpcSketch;
+import org.apache.datasketches.frequencies.FrequentItemsSketch;
+import org.apache.datasketches.frequencies.FrequentLongsSketch;
+import org.apache.datasketches.kll.KllDoublesSketch;
+import org.apache.datasketches.tuple.TupleSketch;
+import org.apache.datasketches.tuple.aninteger.IntegerSummary;
+import org.apache.pinot.segment.local.utils.CustomSerDeUtils;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+
+
+/// Regression test for cross-version serial compatibility of DataSketches objects.
+///
+/// The Base64 fixtures below were generated with datasketches-java 6.2.0 (the version shipped before the 9.x
+/// upgrade) and represent the bytes that exist today in BYTES columns, star-tree aggregated values and
+/// intermediate query results of deployed clusters. Each test asserts that the current datasketches-java version
+/// still deserializes them correctly through the same [ObjectSerDe] instances used on the query path. Do NOT
+/// regenerate these fixtures with a newer datasketches version - that would defeat the purpose of the test.
+///
+/// Fixture recipe (datasketches-java 6.2.0): 1000 distinct integer keys (0..999) per sketch;
+/// frequency sketches received keys `i % 10` / `"s" + (i % 10)`, i.e. 100 occurrences per item.
+public class DataSketchesSerialCompatTest {
+ private static final String THETA_ORDERED_620 =
+ "AgMDAAAazJPoAwAAAACAP9WJ7fK6dCoA3qxr4eHZZQD6dLNMFrFuAMfEgYUTf3MA/z7bSTH6lgBRZwE+7+LdAEu/fu5+QPUA5MdYEprEBgGvXx"
+ + "WmSZJOAQoi27cch18BiGVXN9EKdAF7f+xm7Xh/Afs4eYkTJI8Bub+f6vqeqgF4PUbjfXzTAe+vrs9b3fQBlN7WyKB1KgLR7rWxGbSTAm3z+4hV"
+ + "jZYCc0eibjlNmQKumtEZpHAnA6WQe+U0+W0DuDO1GsNucgMPXRGQ3IixAzk+6xYTzrMD5mI4bj1/0wM4XaDwZjvVA3NiaN5La/wDSWK7kGK2/A"
+ + "Pop71o/guGBJ1D/+JVgaAE4trPn83qvgRFpUczrKzdBO0LTW1SugYF/I4BX1yQKgVTe4ImtqZDBY6nDkifGFgFCsR19Lb1ZAVS7BlJHS1oBYkC"
+ + "PEtPTGkFKZIHncBZbwUV+X3LvYahBSWWeWyGPrIF98550oN/yQUWUnbRYWjiBWqpyt4kCu0FofaJYj2PBAb7irflPvt7BtL6/VNWDrUG20AN7O"
+ + "dQzQZdvZ9541UyB67L0A3zAFYHxqPHz0/0jgeYe3jOVSmtBzwvexgKD7sHt85zc3q/2Af5pTFJsvfhB2MxmlcrLeYHjpWNEGFd8Acfco3xbYEO"
+ + "CAtZYHNAkzAIQN4u4cnbPQiSo7xw/Jc/CNqGewlfR0UIaYu5kbhoVwjcP59UDJlaCKkPeAxQUHcI9azX1bj9ewj6aflZPKKDCKD9BmF25Y4I5g"
+ + "vHbkmvpQgc9dXFcQbECPOF0Rdl9sUIvhEek4VG/giQ4d0UI8smCUNs8OgU4DcJGKUmy7vFXQmowdotX/qHCZsmPsrMW5cJb4+pYQUerQmjW5QX"
+ + "zU6uCaG6RKGKXOIJtGWaAgpH9AkoIIQlgHxRCgUDwCUQecIKsh6LHEUlSAthTs01Qn9vCyAZ3mW8Y54LJr+FKnSHwAtNETqMK8/yC9afhNy05h"
+ + "UMO/Rz/J53JQyyb0VgFxA2DM3dMbQZioAMcheLPeVuiwyAoBdQMjm0DKwoeQfM1s8M/cbFSAae0QzkkNLibxfWDClzdbhQdeEMg7kyzPlbfw2v"
+ + "CIHtp9OhDcoh6wMdmb0Ni4I9PgVtzg1//pnso1sZDrgh5FDua2YOsmILovQE1Q5qkYsLC+csD61nsbOVWF8PlOXnMZ2LYA+NeNmPYKaCD1uT+U"
+ + "gl2ocPQ1tVNu7wmw9ysJ9FRMecDyT6oqAzR6IPD/dgftE9wQ+htuPRggTcD42ivBhKh+8P+XfHDAk6ChButddTTU0NEMQTUCbcFRcQkImrOXHA"
+ + "GBA7J2vj1fwtEP4WIRP7mLwQ1yHhBW4Z1hCaWlVHG+7kELx6uhb5O/kQWr90ON8mERFvK7zg+WtdEeiv/P3Yb5URAHhHNuAfnxFeRAcXahOiEY"
+ + "NFfaKZ6a4RkZ31XPZvuhFsRRrZVkDAEalB7ZfhP9URCm0UCRZm1RF2xpAbu5ADEgRkxiW4NQkSa8bONx2CERKg/nAl1BI3EveHBSgW60kSOq80"
+ + "JUB8gRLPVVB60gaTEhlki4wn2JQSIzyZ51PjvxIunwwS0BblEl9nG9hORA8T8yh3Xow4HxNFLnTwuj2PE6ZTbV6AZqkT+ZqeZAHysBM/XDSKh6"
+ + "LHE9UKUugeKMoTTUWpPUIu9hMnT2g3yAUJFMsxC4dlBgoU53Ak5KbbFxS3/6nmHBcbFLUKIyFOnykUkJv3cVxkLRTXxOhkyD1UFAOtk/BdoXEU"
+ + "vTJzckaRzBTd/BiVlFndFF4QKeEI5t8UrPUwSzu79RToAvoztij4FJKXgjbltPoUT0brs+vVIBUbTNFJlZhSFTBAtV3Ww2YV6ex3T6iAfhXOJ5"
+ + "i4Ws9/FcoDYarnM58VE+nHY7M+nxVDXrfaZJezFWatbFgPltAVjO2EWqFu7BX61riVr+r4FcpwHyrrjiEWcXxrntjZZxbhuOfIN0CjFt9mfzCC"
+ + "gqkWq/iLKMymqRZTUj9v2ImrFtFeoEKNnOsWCxcQ+TJ2ABePjGGL0NMRFwP65QUNbTIXlbO1CggRPBdQhFhOeizQF60ExIWycdEXdZk2tVcy6B"
+ + "cO7PR2JzoOGGafQy1udYoYUD1Coya1qxi7EayzlbvCGPMaSqnZaN4Yn+ztiGyA/Bh526OgaHA/GZWci/6z1IoZ4ljTRiePtRnJIN6oJn6+GVlU"
+ + "XH2VZNYZzCQhCqlc3RnWfd5ZaxHhGY9mvalLU+wZKtvQXYFv+RnZYQ82iDQnGligfvAgpUMas8FBsXZNehpu+20XlTisGuScec0cU84aSeUjzE"
+ + "C72BqsW+JmH57ZGrenx3ITWdsavbctHIN3VRsNkVTKht+AG6YM04BTkpMbo1AJ1GLPwxtwQqOPE6fGG5ujhBEG3ckbP16bJGSm1huOptONUTTl"
+ + "G/l5qTyRifYbUjWcO3Es/Bt84iRSDh1MHLlg7c31zF0cVZ+xc1+udBxP8k3gvAyuHCcpvfRIDtUcYfm1Dgrg5RwFCie+dYIGHfvr0BRSGQgdTu"
+ + "7oUlLYCx3npLnMH3ZrHW+xaen9hnIdeTwuTqHVix1wVtdQdUCfHQkCvUGwWKQdEIYEJOXtwB3qITeGIdk+HmgH9b05cVkebakWvEpmYR6qPAKn"
+ + "d/tkHsWQrmZW9XseaS4kUMz7lB7Dl/wSgXCdHioDFntvuLYeS6RB0D6x6h6Kq4i29x0JH1GyvHambw0fAUc5g+GYUx/lAXZhZZdZH9ogcq9w7W"
+ + "YfuGJ6Xm3AZx8qGIvOQTtxHyOlWzga/XQfBhNdn2kpiR+F1iC77N/BHwJeoci3h8kfuke1VGTg/h9mx6sYkegpIJr7KY1majggZUavqhOkQSDl"
+ + "a2HuyIBEIKvoupWxdEggAW4pMD2oaiBVcQ0ikReoID663KNvb7YgepozehV9/iAfSCTacPgXISQKPvxyxUAhmphx/uQ/QSF0aZQGnF9lIbbyjo"
+ + "lPX3IhPG/YJMzkdiHGJNYSRf5+IY8fDSIzNZUh9FDJp1rK2SFwV1aKXPoTIjFbuxNXQhQirFCasegVHiJQ5yUMpQktIhrRMAuZjC8iitRk/Ts1"
+ + "TCLAoq2UCA5aIt16ta8fH2EioMTR912wZyI1oJ1kb7LkItOIKA6TPvwixshM3og9KyPqS4qRYc5gI1D6wBNbJWkjbMx+3shZpiMaNYd8IgfoIx"
+ + "ud9uE8/fAjjE+tMcIeCCTfTr/rOSE3JJb1Ho147j4kVLPec0PyTyR+Cu/6YX1VJCijcJ7ffF8k6DjBeEMYmSTPv4fCNfO5JPD1dTMn1BslIlG8"
+ + "idsmHSXRzpB7mVgmJbQZMYD0ciclDCcFhh8pdiW426UaAMyJJSXrwmIuDqYl95b71QlJqiWVlFelpz64Jc4RCXwy+QkmsVwEBcZvGyaM/62Hnx"
+ + "8eJhXgItfNUycmTWn11tWLKiZzlmxobBg2Jsjtqwp7iDomjNDJhkzJOiYuy+tamBGAJsqsqRfTLZsmcP+Av8uYwSYDISVUhMrYJjleZ6KA1hon"
+ + "b/3c9FyuLiepXadS1oY1J7e/QnmVNkoneGVzrfuJSydOB9MYSH9vJ196fBWCYIYnF42JbY/ckifT8jEaYsOnJ/DsvyC0xK0nCKDbpGhfryfJQ5"
+ + "tIB9LSJ5BUCEa4Uyooy10wLfbEPChYGWTZCw+EKOhCIaDpIcYovWX7U8sm4SjBKlFkVBnqKCPr2aJeGB0p1blD14YpNimZNt43R7I/KfD2pOEV"
+ + "nX0psolb489juSnwmCwi+ZO/Ke/tjLF2O9Iplym7IcuW7inpibPEJykPKlIqyiZ1/XMqcpcqKwQlfSo5IEvBRWfFKvn+HCvEf+gqustp16fyIi"
+ + "udi/lovrc2KxJorLksZkcrD8Inw9cVTCuefLnsoBp0KyvMP32+EYQrmKEvKfUslyvRtnKyfRm8K4DCsYdl2s4rpeevYnkt4CtjxfJQV18YLBi6"
+ + "bxyHmx8s71Z8MD9XNiwuaIb6r2NNLDfrZzIbio4sHXD3CaFBlCy2/7KauJGmLFoAnkI4NrIsXhtllRo+uiyrAiKZ97S+LMvWYUISwsMsdvSeyE"
+ + "oSyywI8K3c5voSLeBBcEzihjstgUGvejpIPS0pe7UvUVZOLWqv35lcaGgtmtQYKDOmoS0li/n/s6tALmrY4+4zPUouZ9D3YBMITC6zGCJ64fp4"
+ + "LsCVLvC1NMgujKZiWOmnzS75VfilcSXULmheuQafVPEuB3A90e2WFi+sdwTsHmklL1f3YKWmUDov9lIjgTMLVy+awwugCmRjLyor2CD1YeIvOe"
+ + "EWLODS/y95PEu02iZLMF4PgvG7QlkwUkY+Iw/PgjBtB243mWy0MJkSsQiFE7cwRQsEQ9NgwDCnZBNlpo/XMNZbZmz9jCYxxViylFsjKDHfjMDm"
+ + "RUdLMfCYAS+9blUxTyUsBqmjVjEHRN/7yM9gMe1x6ia4hYQxyuEQukCGkjG6wMcVau2xMSAY66sUorMxQksyUC9v/zHbjOc75owOMtQoUNm08h"
+ + "oy13Gh4GtyXTJgaJOK84d+MmMUN6OcOtQyPqG4wYz8AjN/cfd5DtcxM5wVfbRdSToz+9i9G1VkOjMJBTE9AeRPM+GQPauRJVozPhW8D8fmiTNI"
+ + "1wVVIf6TM7zDm1up0pkzao9+4Dp6pjOWftXgVcDOM4al3+bnYNQzVqqm2zlrFDTd5IBItcImNFR9bncmOKY0e7GQCyhtxTSSdQWmSd7fNF9MzN"
+ + "VvIug0wB51shtp+DThU3uOJnf4NJ296gleQRc1oYsubGdLIDWms+9EGMRLNXrtNE9uVlI1RuLEMqskVTUiDXHyJYOWNflQ/WrBxaQ1hwkFtvRB"
+ + "pzUj31PXSD6wNRNBl4NTusU1G3YxWo5V/zVFMTSHfvgNNlRQ4WHpl3I2zLRfVZ9pxjbXomAmSW/QNj3hFkwV5dE2oLghJQQ94TZCIn4/fWAfN9"
+ + "dobd7qjzw3eiurFapVXzehDaUC5jBqNwz/V0oJ1mo30aL9LGsVdTcBmUCoozF8NykxXTSeuYE3DAUj0xPLjTcVOJMvHdOuN2NdwnrlCJY4VVwW"
+ + "97pnIDnELlms8pshOaDjWFFZQVY5Ckunw8njYTlz94MfE9uHOXa7JYfcvaY5y4koqzQa1zkD7R8Vu2D4OZ00V45JZi06QLT7eYpAQTrax0T96f"
+ + "tSOlz2twbsOmM6jBgv++XzhDpXHsScr0GXOjyPiZZU+7w6gIzs6Zto3jqe2lYkR8jqOsa1RrYquhM7iMJKUMLkFzvCNObiWYQdO8b/7p64Pzc7"
+ + "tqHFx6hBOTtx/mpsjsBGO8BXGaKG0GM7kNQMF+q0oTvVxvryMWXNOw8U6SPD4uI7ADhR1b+J+TvA36U5DYD7O5xFlEVQCwk8oOtrKoKwHjyX0U"
+ + "W/4lsuPBajAtfuZjw8PsCxqj4lcTxrPBo3zAV2PBoXX0lgX4c84qn+6rtpoDwOw0G5NWeoPI2Bn5T5Bf48BmIkToTIBj2LM/iJ2o0HPVgZ2+S4"
+ + "1hw9D5dOMJJETj1cq1RAEz15PWIjuyHvSbM9/Mb7B+KjxD30QraTrwnHPXchJfT9MRU+25cE8XrbIj7KUCfUTu4/PlAfAKaZEVk+cBINyavtWj"
+ + "5HTiGtBNigPlh6mT3IV7E+32pMOKf+xD5Yz4n447zjPvWQU3wXTPk+yWWfUojJAz8AIfRd3YsXP8K8Xf/IekI/cIcvm19DUj9BHnjCrCFgP2f4"
+ + "rfV42Ys/t2Wl29aNjT9QPfIy1EOmP9pYlMZjZrI/yHsJociK+T+BWF5gIGonQGbX0q7m12BAo/XzlxOLkkA0CPX4kc+7QIGECRDLOu5ArcE6ZV"
+ + "C87kCC9/Gqx1FMQdd51rNf/XBBTcSb//B6tUEbU+X7i4bUQc/FtTlDANpB47D+ZsLR6EHfMvsIPswNQuJbt4LyiCRCVKyZSIwbOULbBbv+Oblb"
+ + "QgEJA0qko3xCFWQQf49LtkLReU6ft2HHQlBn+cyoMeJC6aaXgHaY+EJhWBwM0oX9Ql6OijkkUQBDMQ5r/BI6E0P/GRv2XN5gQ5QSEbCAtohDRs"
+ + "EO3QyCnkPpeF47IZnsQ6emXwDuxX1EebCYWepgs0RcLT2J8wS4RFiJbcGEhrlE4e4o4grYzUSEVO9y4iHjRMHHjmWp+/pEfxC5fQuLHkWfZhBC"
+ + "Gc8mRf+6+ClB/3NF/enlu9v8fUU5yt5yMeV+RVSNknmTtqhFUfU2TY33ykXUlB1n/HHMRecrBHcmgiFGnpqOJbftM0ZgQDaN3aRERrqEmS6a2m"
+ + "xG1gnyME7ThUbEh+Q5oxC/RgyqXuWJ8O5GorHnc716C0cSL9P0iRaMR7nwFrkO/JhHtjwv3MoZ0keagaEl/jv5R6m2F0IpjgFILbcQXk9zE0gN"
+ + "Xujm1xlASBOU9X9uw0pIt2IaKY0JS0iTWdN3axB/SHk+DjkrXMxIqr2jwfsmC0lnA4i2yDMoSWQ9Em6L1ztJYgxqzQqSYknO4TLiQ3KvSR4U+E"
+ + "1yjNRJaT7GWk+zDUo4Q5NSXSVISoHTWeeb9Y9KuXwEMRs6kUoE4rjHk+vkSm20D9OBch1LIAujP4N/Xkv/qW8Me85+S2mUiVB2FpNLNXTFEwtI"
+ + "qEt0UsdRNtOrS6Z+ZQMsvLhLrFucjSmeMkxIIUgPILQyTBsheG9wzD1MSXbRXGekdEzUzTpi3zWbTLzvMl6Kh6xM1GBiKtOOsExboEzctJLnTE"
+ + "/iUX9fW+lM/OYWNiKZQk3W4/l0Zy5nTSYDWJkL8HhNNz0wvJnlm02Rh2/kx1uuTYn83DDll9JNEO5POfMUI06XVp+Aiz9ETrkGA71JAY1Obq0A"
+ + "Z75M6E7jDDUWyQruTr3Go78gl/FOZ1YkGQANSk8U/frdos9WT8QghgMWintPEdyOkl/tuE9T5q6zh/fCT1nUVw3Q7/NPFLL6R7Zr+k+Rw/+Ni4"
+ + "gJUDO1luENbEhQx2FY90DES1CZcvITs05zUJiirH3HgJBQTsb6GkfbrVCwyeikgeAQUd+KqPdgBhxRNWZ/6Js3QFGk+GbIDahfUVifzq2qXotR"
+ + "GH3DjdJ9zlHa78F/F1PUUeI+ZCw9zORRdVc0DlMb6VH5SiibEi4JUixiJ1jWexVSe/6PI4GOG1IHGbov/psjUr1o87Aqq2xSu9UNeSOa5FJvaN"
+ + "IX9un5UjLMo/ybQypTwOD5osfFulMEAbRA5N/KUxi2gY/3XMxTDm9jF7Go8lMvQnPsdhX/U5Jn58Q/AgNUy+AsnsOpE1RAcfgWcYomVNUnWDsu"
+ + "TVlUsAoUO0XEWlQbtOyqqHNiVKDtOpBQap5U/o9J+ItCrlSgEpTdH1K4VPms7Qm9GABV1J3aP2evIVUYPUjILtA7VbMfQa9benZVTCPPxId5rl"
+ + "VSh541nnS9VWqVviro2ytWQ3sAzzjSMFaTwE1ivTVdVvMqN4kNdGNWqXU6yOEwe1aJxwgkAAXFVrL364NhSstWXLB85FYj11Y7Af+htkPbVm8H"
+ + "2FUfaQFX7vW0tqwTDle2iBlmUBIQV120WTtBsxNXv9aVWlxVH1dZ2E/hQm48VzHDh8DqnGdXy18UNiUraFdAz8KxxqyMVxHWaEb0OplXBaRrw3"
+ + "IvuVfqRUYEZq/NV++pFu6FCedXELNxZUAC9ldRIXh3d4BmWBclEIxuL95YvrK87ci441jinrROoM4HWVSndiDOGiFZfwsWGSJ0OVm5udU+Lrja"
+ + "WWYhQe9OgTham+KZzk6UaVp5x4W3sT53WuHakiK+rohaeN2Va4AajVp+zZjQ3v2+WkoceR6MttVad16Y3BUEHVtU2ESgvW0eW3JIT4bFyidb02"
+ + "w2ffYwQ1ukdjV9asaMW4sij8TzfZJbUUTXSxT5m1tbV3jNDAuuW0bUezw9asVbjQNh4Dmh5Vu8I7SEfvjrWzS2+1xO4P5b/bRhCXnaQlxoPYMr"
+ + "KQZrXG3iqdZfxnZcZBh4lmsLiFwmpHUV9R2IXFp3Hqj73pJc2Up48jnZn1xcBi7IQHe5XEtY4Hpcv7xc8FI6P1k4zVzxyGBs/9PcXAWGRWbqNO"
+ + "FcuxtirpF0Fl2zCExS8ZVIXbpAs8HaBmldSX40gs2mcF01CrUbLkp9XcgK07dUoyZeOp9AD59rSF5yKERHxNxOXtK3VRdU/dNeln94GNUS4V7C"
+ + "MWyPNwP2XhvW83E+wfdeSTrUDYAENF99+2Uujxg8X20uiZ/x7UNfkm8/v8QuaF/llTtknKhoX9a13XTrsGxfE4eZYMw/bV9wgowxkSqQXyPI36"
+ + "bokJxfWqbWLQZupV+VnHg7bcaqX/TjC4NQWdlfCzRUObe56F/nJd9iiEoRYDlJ4BjSUl5gPJZ3OSC6gGDLabF2q5ukYBJcMvvWUq5gp5SvAqP5"
+ + "9GBflFoVj2M1YSOwqWFqcTVhg773Co/jQ2GwNt4NUb11YS+9Xb4GgIVhubT4fXfb6mE9kCVgCiX0YcyFZE48wA1iBpwtzT/QRGIrlw4eer9RYu"
+ + "odhYER1mxi1N7FK/IbwmKXeZVdEmHzYmJmfhc6avZiH2wHCB3qAGPJZPqBWk80Y9DBSDJL4UxjgbzZ1TVUUWP75kFJKCqtY4Bgr3VaBfpjGNin"
+ + "0wOuI2Sxi+8tOzaZZLhku1cMoLVk3AB+582U/mTdRyan41j/ZA5FPGUKkzBlwfKIeHe+NWUxcQmgc1JOZSMMb0XBgoBltNN9Ojr1gWXCmx6fXp"
+ + "axZbFspFHO3+1l7wvSqxnZ/2UAeyO6qOJ9ZjBjxw+Prdpmhq+CKKGzWWdSvNnJeNPRZwT88IM1qwlo0R7D+TT+mGhVMd6302mdaNIOv8+DGqxo"
+ + "a4bBypAaumiGL728xefOaEv7tOSQCiRpqyFVM0L9KWlwA7Tin1YsadqIQVuTsDNpF/IFeXZMV2nu92UVt7dwaRlDVKEtjHFpJtBa0wQtpWl8V8"
+ + "+/pfD7aX5I27YSHwpqsIrwgx+bE2pVosJRiPQmao1rzffcLVRq1fqh+9B9VGodgzwV/oJpanmPvd1N/5BqQ+F1Bzxan2oxVVNZUWy9alTU/J5T"
+ + "GOdqamS4tGs09mru5eaOBgf9alDQI8zG2WprhBiXuuO0dWvsDSVQpU8NbA88wRMtXn9siiactiAfhWxHEn+xm56rbCoZLk7S+7xszWjKsTfODW"
+ + "1fII0E+BtIbe4eAKkWHVht+OTb9Ap8WW0UFMJFBoFabSGi7NfX1mdtGxVZPD9Me20MWdKuC+WrbcU3nQPj6+xtJc3zC1G/UG7z58pHjMdXbmEO"
+ + "RJL6tHtuFjXLFNGcim5aI0hoDruxbpnnpAcNOtVu1sMXHdRS4m64iM6RGLIVbxnHyx748x5vabbp0BxMI2/cn/yx1Ecob9XEQtt6b09vzHs/Oz"
+ + "1lrW+mL5IKfReyb04FtgdKJcpvn2wvpe7FzG9+TVMuCbUvcMV/w2SAI0Fw+wSWVB04UHBAnNyT5x6WcCd88QJ7zZdwimPEo6QFpHDn7Zjoky7K"
+ + "cIW4o4NuBtxwmpBPXadK43ByfbbkOcoocQERWtK/NUJxoXT5pmcpbHFJqpj8ZYyVcc8WtQFoUg5yE7E2hhLfL3LQcoytr72yckyasf1kL79yxX"
+ + "jBCjbv63LuT703Q8LvciEoVStvfwZzn69JeVTSLHMcrvFDkZ1ocyYLbVtqtrZzAGUs5kQbx3N8/Hi0MD8BdB2CduIgoUN0V+l6Cg4jWXR529Pb"
+ + "KW2zdFcJN71SX7Z0h6W6+SUzCXVsyU0wW3xAdXzYxdwuukd1qR7AYu+yt3WFTOgYaWPZdcN73tj1fPN1u3foXfTHD3br53hbDf4ddgY88T4RgB"
+ + "52WaRVDXaAJHbJrHnCSY0odnKw+D/L51p2AsSp14P7dnZRVr/+meOHdr9bN0A59Jd2mFQw2XMdpnb6nu8zTg3GdtuXTffvn9B2IRoC48Zc2nb+"
+ + "9D5ZA/rjdhzAWhaX0id3buTFyYTDSnd3m7/vS0y/dxH28GHIkeN3qfpXL1Nj+XfTjD5cRmD8d2sCDKKYZDN47/RANwSoYng9xdSkgQmCeAQVuQ"
+ + "iizYR4/TqfOxQdAHk35tG4sKQFeRK+gYm9mxV5bMK9GajjG3nsWqj/O20yeRVY1FKeXjt5tiI+M6NOXHnTZJsJBpWJecQ+OMBgopJ5Sr9Qau+n"
+ + "tHmsq08Ee2zDeVHBZMpJuNJ5MwKuHQNa6nkIb14ogYbqeSJhg/GnNO153Py6aCzJIXqFZAcWBokiejq709nf1z569t+QKk2yVHqelkZHzdSDer"
+ + "Y8m0DMOIV6Al2SD33l33pHFxYtIU3xeqw/6tgxGgp7nd5eYJqQF3tyFT1f4DlhexSPcWUvsnF7XqL0UXhVkntCxdV12uiZe53tGzCuqLJ7E3zL"
+ + "43g2MnzZc8WQws9QfFT6IUaqrot8l15DVd8il3xwzOnnR629fLDM0++KlMF84PSL6pmDw3w2U52a/SzFfOSnt0CVOxl9aSlmx+Y/I303Phk3Xm"
+ + "QtfVDsNSqp1UF9qlf5SChhSn3FqxUxdxlhfec9QYcVsZF9qN+AqobWuX1l1KBbsD3AfdlSpqR1R+J9/YQ3xqsW533D8rmNpV3yfTEQdD+vVfl9"
+ + "6pgJOV03MH6/7wCNc4wwftgtI3dLuTV+8N0X8Wt2/n7ctCA0xSVtf2QzddqfWL5/m56n4Vqlw3/3XorjaDXEf/WhA2wR7sR/tHIbsHex13+31B"
+ + "fXKcfff+dPeXlgcvR/";
+
+ private static final String THETA_UNORDERED_620 =
+ "AgMDAAAKzJPoAwAAAACAPwig26RoX68nFeAi181TJyYcwFoWl9IndygghCWAfFEKMEC1XdbDZhU1oJ1kb7LkIjkgS8FFZ8UqPsCxqj4lcTxYoH"
+ + "7wIKVDGloAnkI4NrIsW6BM3LSS50xfII0E+BtIbWBANo3dpERGgGCvdVoF+mOTwE1ivTVdVtyf/LHURyhvuWDtzfXMXRy6QLPB2gZpXcDg+aLH"
+ + "xbpTxCCGAxaKe0/JIN6oJn6+GcvgLJ7DqRNU1GBiKtOOsEzaIHKvcO1mH9tADeznUM0G3AB+582U/mSAoBdQMjm0DAAh9F3dixc/AyElVITK2C"
+ + "YEAbRA5N/KUxNBl4NTusU1GyF4b3DMPUwiYYPxpzTtebrAxxVq7bExOeEWLODS/y87Af+htkPbVj3hFkwV5dE2PqG4wYz8AjND4XUHPFqfakbB"
+ + "Dt0Mgp5DSCFIDyC0MkxRIXh3d4BmWGYhQe9OgThadyEl9P0xFT6BQa96Okg9LY2Bn5T5Bf48kOHdFCPLJgmYoS8p9SyXK5qBoSX+O/lHUcFkyk"
+ + "m40nmowdotX/qHCalB7ZfhP9URqyFVM0L9KWmtwTplULzuQLPBQbF2TXoatqHFx6hBOTu4IeRQ7mtmDsdhWPdAxEtQyuEQukCGkjHO4TLiQ3Kv"
+ + "SdDBSDJL4Uxj1yHhBW4Z1hDZYQ82iDQnGuBBcEzihjst5QF2YWWXWR/qITeGIdk+HvWhA2wR7sR/BOK4x5Pr5EoGYiROhMgGPQkCvUGwWKQdCi"
+ + "LbtxyHXwEPwifD1xVMKx2CduIgoUN0IaLs19fWZ20sYidY1nsVUi9Cc+x2Ff9TMwKuHQNa6nlCIn4/fWAfN0bixDKrJFU1SWK7kGK2/ANP4lF/"
+ + "X1vpTFWiwlGI9CZqXqL0UXhVkntrAgyimGQzeGzCvRmo4xt5beKp1l/GdlxwQqOPE6fGG3NiaN5La/wDfOIkUg4dTBzKIesDHZm9DYDCsYdl2s"
+ + "4riMJKUMLkFzuJAjxLT0xpBYuCPT4Fbc4NjaK8GEqH7w+Yoqx9x4CQUJvimc5OlGlaqwIimfe0viyyYgui9ATVDrYiPjOjTlx5t2IaKY0JS0i4"
+ + "YnpebcBnH8CirZQIDloi0aL9LGsVdTfXomAmSW/QNuZiOG49f9MD6EIhoOkhxij0QraTrwnHPQUDwCUQecIKDsNBuTVnqDwWowLX7mY8PBlDVK"
+ + "EtjHFpHYM8Ff6CaWomA1iZC/B4TSijcJ7ffF8kKgMWe2+4th4wY8cPj63aZjHDh8DqnGdXOEOTUl0lSEpwgowxkSqQX0wjz8SHea5VWiNIaA67"
+ + "sW5iI7sh70mzPWcDiLbIMyhJcAO04p9WLGmLIo/E832SW+gC+jO2KPgUimPEo6QFpHCNA2HgOaHlW5HD/42LiAlQkqO8cPyXPwiawwugCmRjL5"
+ + "ujhBEG3ckbnUP/4lWBoASg41hRWUFWObwjtIR++OtbxqPHz0/0jgfJQ5tIB9LSJ8oDYarnM58V1sMXHdRS4m704wuDUFnZXwLEqdeD+3Z2BGTG"
+ + "Jbg1CRIFpGvDci+5VwdE3/vIz2AxCsR19Lb1ZAUVZBB/j0u2Qhlki4wn2JQSJqR1FfUdiFxLpEHQPrHqHk3Em//werVBUIRYTnos0BdRRNdLFP"
+ + "mbW1mkVQ12gCR2XkQHF2oTohFqZLi0azT2am7kxcmEw0p31uP5dGcuZ028w5tbqdKZM4GECRDLOu5AhWQHFgaJInqgxNH3XbBnIqdkE2Wmj9cw"
+ + "rQTEhbJx0Re4ZLtXDKC1ZLqEmS6a2mxGxiTWEkX+fiHHxIGFE39zAMlk+oFaTzRjzCQhCqlc3RnTZJsJBpWJedXEQtt6b09v18ToZMg9VBTd5I"
+ + "BItcImNOekucwfdmsd+OTb9Ap8WW37BJZUHThQcP2EN8arFud9AGUs5kQbx3MJBTE9AeRPMwwFI9MTy403DkU8ZQqTMGUXJRCMbi/eWBilJsu7"
+ + "xV0JI6VbOBr9dB89xdSkgQmCeELF1XXa6Jl7RaVHM6ys3QRJ5SPMQLvYGk1FqT1CLvYTTgW2B0olym9PJSwGqaNWMWPF8lBXXxgsbEUa2VZAwB"
+ + "F4ZXOt+4lLJ4NFfaKZ6a4RhqXf5udg1DOHpbr5JTMJdYhlVzfRCnQBlOXnMZ2LYA+cRZRFUAsJPLRlmgIKR/QJt2Wl29aNjT+9ZftTyybhKMll"
+ + "n1KIyQM/zIVkTjzADWLPxbU5QwDaQdsFu/45uVtC5yXfYohKEWDqRUYEZq/NV+7l5o4GB/1q84XRF2X2xQj5pTFJsvfhBwWGRWbqNOFcEIYEJO"
+ + "XtwB01Zn/omzdAUTfm0biwpAV5Tsb6GkfbrVBPRuuz69UgFVJGPiMPz4IwU+aus4f3wk9aptYtBm6lX1wGLshAd7lcYmZ+Fzpq9mJlRq+qE6RB"
+ + "IGvGzjcdghESdsaQG7uQAxJrhsHKkBq6aIomnLYgH4VsjKZiWOmnzS6OptONUTTlG49mvalLU+wZmyY+ysxblwmfZhBCGc8mRaemXwDuxX1EuQ"
+ + "YDvUkBjU69xqO/IJfxTtXG+vIxZc072oZ7CV9HRQjfZn8wgoKpFumml4B2mPhC++ZBSSgqrWP85hY2IplCTf3GxUgGntEMAUc5g+GYUx8MJwWG"
+ + "Hyl2JROHmWDMP21fGcfLHvjzHm87J2vj1fwtEE4H0xhIf28nUGf5zKgx4kJRZwE+7+LdAFKHnjWedL1VVKd2IM4aIVlfZxvYTkQPE2bHqxiR6C"
+ + "kgaAf1vTlxWR5tB243mWy0MG8H2FUfaQFXcIcvm19DUj9zR6JuOU2ZAnnHhbexPndaiccIJAAFxVaOpw5InxhYBZGHb+THW65NkmfnxD8CA1SZ"
+ + "56QHDTrVbqXnr2J5LeArrWexs5VYXw9Q5yUMpQktIrenx3ITWdsa/Mb7B+KjxD26R7VUZOD+H8HHjmWp+/pExIfkOaMQv0bOJ5i4Ws9/FdUnWD"
+ + "suTVlU2sdE/en7UjrdRyan41j/ZOTHWBKaxAYB6Ke9aP4LhgTr53hbDf4ddvPnykeMx1du94cFKBbrSRISaKy5LGZHKx9IJNpw+BchIShVK29/"
+ + "BnMjyN+m6JCcXy5ohvqvY00sNAj1+JHPu0Dkp7dAlTsZfWBok4rzh34yb2jSF/bp+VJyKERHxNxOXn5I27YSHwpqq+i6lbF0SCCsKHkHzNbPDK"
+ + "8Ige2n06ENswhMUvGVSF22iBlmUBIQV7iIzpEYshVvvWjzsCqrbFLGyEzeiD0rI81oyrE3zg1t04goDpM+/CLUKFDZtPIaMtdobd7qjzw32ohB"
+ + "W5OwM2nxyGBs/9PcXPMod16MOB8TAQkDSqSjfEIT6cdjsz6fFScpvfRIDtUcOUngGNJSXmBNafXW1YsqJlfpegoOI1l0WIltwYSGuURySE+Gxc"
+ + "onW2kpZsfmPyN9aqnK3iQK7QVsyU0wW3xAdW2pFrxKZmEedGmUBpxfZSGHCQW29EGnNZCJqzlxwBgQlym7IcuW7imwyeikgeAQUbKJW+PPY7kp"
+ + "Vwk3vVJftnTLiSirNBrXOdWJ7fK6dCoA1gnyME7ThUbLabF2q5ukYOmJs8QnKQ8q76kW7oUJ51f6aflZPKKDCP3p5bvb/H1F/6lvDHvOfksFCi"
+ + "e+dYIGHQyqXuWJ8O5GJAo+/HLFQCE1CrUbLkp9XTnK3nIx5X5FSaqY/GWMlXFSKsomdf1zKlaqpts5axQ0fgrv+mF9VSSwChQ7RcRaVLUKIyFO"
+ + "nykUsIrwgx+bE2rBKlFkVBnqKMgK07dUoyZe4qn+6rtpoDzVClLoHijKE9lKePI52Z9c32pMOKf+xD7zKjeJDXRjVvlKKJsSLglS+4q35T77ew"
+ + "YKS6fDyeNhOSALoz+Df15LI+vZol4YHSkl68JiLg6mJSYLbVtqtrZzKivYIPVh4i8uy+tamBGAJjfrZzIbio4sQksyUC9v/zFFCwRD02DAMFyr"
+ + "VEATPXk934qo92AGHFFpi7mRuGhXCG8rvOD5a10ReiurFapVXzd/CxYZInQ5WYqriLb3HQkfjWvN99wtVGqdi/lovrc2K6DrayqCsB48oYsubG"
+ + "dLIDWsq08Ee2zDea7L0A3zAFYHsYvvLTs2mWS6y2nXp/IiKyWL+f+zq0AuxasVMXcZYX3la2HuyIBEIOYLx25Jr6UI5ysEdyaCIUbqS4qRYc5g"
+ + "I+0LTW1SugYF7wvSqxnZ/2X769AUUhkIHQ7s9HYnOg4YG0zRSZWYUhUfbAcIHeoAYyMMb0XBgoBlK8w/fb4RhCsyzKP8m0MqU0Ns8OgU4DcJUO"
+ + "w1KqnVQX1S7BlJHS1oBVSsmUiMGzlCX0zM1W8i6DRiDGrNCpJiSWzMft7IWaYjcMzp50etvXyAjOzpm2jeOoVM6BhpY9l1j4xhi9DTERefbC+l"
+ + "7sXMb6YM04BTkpMbsMzT74qUwXyxbKRRzt/tZcmsecJJjSh2yqypF9MtmybTjD5cRmD8d9uM5zvmjA4y3qxr4eHZZQDfjMDmRUdLMeMMNRbJCu"
+ + "5O6ex3T6iAfhXw7L8gtMStJ/Ws19W4/XsI+aztCb0YAFUD7R8Vu2D4OQptFAkWZtURF42JbY/ckiciDXHyJYOWNSXN8wtRv1Bu02w2ffYwQ1sD"
+ + "rZPwXaFxFFSNknmTtqhFXC09ifMEuERmrWxYD5bQFW6tAGe+TOhOeu00T25WUjV+zZjQ3v2+WozthFqhbuwVne0bMK6osnug7TqQUGqeVKENpQ"
+ + "LmMGo3fk1TLgm1L3DI7asKe4g6JtTNOmLfNZtM2C0jd0u5NX7n7Zjoky7KcOwNJVClTw1s7+2MsXY70ikBbikwPahqIBDuTznzFCNOHK7xQ5Gd"
+ + "aHMxDmv8EjoTQ0UudPC6PY8TR04hrQTYoD5O7uhSUtgLHZ/s7YhsgPwYXo6KOSRRAENhTs01Qn9vC2kuJFDM+5QebS6Jn/HtQ19hDkSS+rR7br"
+ + "fOc3N6v9gHxC5ZrPKbITnRzpB7mVgmJdIOv8+DGqxo306/6zkhNyTh7ijiCtjNRPfOedKDf8kF/I4BX1yQKgUIb14ogYbqeQ5vYxexqPJTEi/T"
+ + "9IkWjEcUj3FlL7JxeydPaDfIBQkU0e61sRm0kwI6rzQlQHyBEjxv2CTM5HYhQM/CscasjFdYz4n447zjPl4PgvG7Qlkwaq/fmVxoaC1vj6lhBR"
+ + "6tCXmPvd1N/5Bqhi+9vMXnzmiMT60xwh4IJJJvP7/ELmhfn69JeVTSLHOmL5IKfReyb6kPeAxQUHcIhq+CKKGzWWeyb0VgFxA2DLzvMl6Kh6xM"
+ + "v+8AjXOMMH48j4mWVPu8OtrvwX8XU9RR5095eWBy9H/or/z92G+VEe5PvTdDwu9yPC97GAoPuwf+j0n4i0KuVAdwPdHtlhYvCPCt3Ob6Ei0dcP"
+ + "cJoUGULCOwqWFqcTVhJtBa0wQtpWkxEHQ/r1X5fT2QJWAKJfRhUNAjzMbZamtqj37gOnqmM1RQ4WHpl3I2XLB85FYj11ZeECnhCObfFGfQ92AT"
+ + "CEwucrD4P8vnWnZ5sJhZ6mCzRH8QuX0Lix5FjNDJhkzJOiaakE9dp0rjcKNQCdRiz8MbpZB75TT5bQOsUJqx6BUeIrnwFrkO/JhHxZCuZlb1ex"
+ + "7KUCfUTu4/Pu+vrs9b3fQB4ZA9q5ElWjPjsP5mwtHoQeSQ0uJvF9YM53Ak5KbbFxT0UMmnWsrZIfWQU3wXTPk++VD9asHFpDUBEVrSvzVCcQ2R"
+ + "VMqG34AbE7E2hhLfL3Ia0TALmYwvIiJRvInbJh0lKTFdNJ65gTcxcQmgc1JOZUBx+BZxiiZURTE0h374DTZNETqMK8/yC1VxDSKRF6ggYzGaVy"
+ + "st5gdqkYsLC+csD3Kwn0VEx5wPb7Fp6f2Gch17sZALKG3FNH9x93kO1zEzl9FFv+JbLjyisedzvXoLR7sRrLOVu8IYvhEek4VG/gjKcB8q644h"
+ + "FsIxbI83A/ZeyzELh2UGChTOEQl8MvkJJtdxoeBrcl0y7XHqJriFhDEUsvpHtmv6TxZSdtFhaOIFF/IFeXZMV2kfco3xbYEOCCmSB53AWW8FVT"
+ + "Het9NpnWhHEn+xm56rbE/yTeC8DK4cUbK8dqZvDR9TUj9v2ImrFnASDcmr7Vo+dFLHUTbTq0uUEhGwgLaIQ5kSsQiFE7cwoBKU3R9SuFS0chuw"
+ + "d7HXf7byjolPX3IhvTJzckaRzBS+srztyLjjWJly8hOzTnNQwfKIeHe+NWXD8rmNpV3yfdByjK2vvbJy0/IxGmLDpyfZUqakdUfifd8y+wg+zA"
+ + "1C8FI6P1k4zVz2UiOBMwtXLwYTXZ9pKYkfELNxZUAC9lcbU+X7i4bUQSlzdbhQdeEMNlOdmv0sxXxUs95zQ/JPJFuT+Ugl2ocPZDN12p9Yvn9t"
+ + "8/uIVY2WAoHTWeeb9Y9KizP4idqNBz2Vs7UKCBE8F6az70QYxEs1tNN9Ojr1gWW4M7Uaw25yA8QTUCbcFRcQ2XPFkMLPUHzhU3uOJnf4NAs0VD"
+ + "m3uehfDxTpI8Pi4jsTlPV/bsNKSBQUwkUGgVptG7TsqqhzYlQeFPhNcozUSTV0xRMLSKhLO/Rz/J53JQxAtPt5ikBBOkbUezw9asVbVNT8nlMY"
+ + "52pZVFx9lWTWGV20WTtBsxNXX5RaFY9jNWFjFDejnDrUMmXUoFuwPcB9aZSJUHYWk0tttA/TgXIdS3b0nshKEsssplNtXoBmqROEVO9y4iHjRI"
+ + "rUZP07NUwikNQMF+q0oTuVlFelpz64JZhUMNlzHaZ2mtQYKDOmoS2dNFeOSWYtOqF0+aZnKWxxp5SvAqP59GC31BfXKcfff7m0+H132+phwjTm"
+ + "4lmEHTvMtF9Vn2nGNtSUHWf8ccxFWdRXDdDv80/ctCA0xSVtf+D0i+qZg8N87/RANwSoYnj6dLNMFrFuAP20YQl52kJc/vQ+WQP643YEFbkIos"
+ + "2EeJBUCEa4UyooFjXLFNGcim4aNYd8IgfoIxsVWTw/THttHPXVxXEGxAgxVVNZUWy9ajO1luENbEhQPhW8D8fmiTNR9TZNjffKRVI1nDtxLPwb"
+ + "apW+KujbK1ZutddTTU0NEHIVPV/gOWF7jpWNEGFd8AeSdQWmSd7fNJb1Ho147j4knBV9tF1JOjOj9fOXE4uSQKl1OsjhMHtWrPUwSzu79RS71Q"
+ + "15I5rkUsCVLvC1NMguxrVGtiq6EzvPVVB60gaTEta13XTrsGxf5ZU7ZJyoaF/u9bS2rBMOV/D1dTMn1Bsl+VX4pXEl1C4R9vBhyJHjdxi2gY/3"
+ + "XMxTG9bzcT7B914llnlshj6yBRt2MVqOVf81NLb7XE7g/ls8lnc5ILqAYEl20VxnpHRMUVa//pnjh3Zc9rcG7DpjOmdWJBkADUpPabbp0BxMI2"
+ + "9wVtdQdUCfHXOWbGhsGDYmhdYgu+zfwR+XVp+Aiz9ETpk23jdHsj8pnpZGR83Ug3qhtuPRggTcD6R2NX1qxoxbqbYXQimOAUiwNt4NUb11Yb/W"
+ + "lVpcVR9Xy9ZhQhLCwyzPFrUBaFIOctG2crJ9Gbwr71Z8MD9XNizw9qThFZ19KfeW+9UJSaol+ta4la/q+BX+FiET+5i8EAsXEPkydgAXD5dOMJ"
+ + "JETj0aF19JYF+HPA/3YH7RPcEPK5cOHnq/UWIttxBeT3MTSKH2iWI9jwQGRxcWLSFN8XpI1wVVIf6TM1f3YKWmUDovWnceqPveklxbV3jNDAuu"
+ + "W2bX0q7m12BAcFdWilz6EyJylyorBCV9KnP3gx8T24c5dVc0DlMb6VF8V8+/pfD7aYL38arHUUxBkpeCNuW0+hSqV/lIKGFKfax3BOweaSUvsv"
+ + "frg2FKy1YR1mhG9DqZV7t36F30xw92vbctHIN3VRvAVxmihtBjO8OX/BKBcJ0exTedA+Pr7G3St1UXVP3TXtuXBPF62yI+7vdlFbe3cGn5d8cM"
+ + "CToKEAA4UdW/ifk7FTiTLx3TrjcY2KfTA64jZCAY66sUorMxcheLPeVuiwwqGIvOQTtxH0tY4Hpcv7xcVNhEoL1tHltZ2E/hQm48VxVY1FKeXj"
+ + "t5YVgcDNKF/UJkGHiWawuIXGf4rfV42Ys/atjj7jM9Si4AeEc24B+fEXzYxdwuukd1gVheYCBqJ0CEGJe647R1a4W4o4NuBtxwjBgv++XzhDqN"
+ + "eNmPYKaCD5qYcf7kP0EhoLghJQQ94Tak+GbIDahfUav4iyjMpqkWsxgieuH6eC7bl03375/QdsV4wQo27+ty2liUxmNmsj/huOfIN0CjFuJY00"
+ + "Ynj7UZ6DjBeEMYmSTpeF47IZnsQ+qYCTldNzB+8JgsIvmTvyn7OHmJEySPAQGZQKijMXw3Bxm6L/6bI1LwmAEvvW5VMQtZYHNAkzAIDFnSrgvl"
+ + "q23FWLKUWyMoMRX5fcu9hqEFIBneZbxjngsqGS5O0vu8bFgZ2+S41hw9Yfm1Dgrg5Rx1mTa1VzLoF4O5Msz5W38Nk1nTd2sQf0iXeZVdEmHzYl"
+ + "gZZNkLD4QotBkxgPRyJyW5udU+LrjaWdF5Tp+3YcdC1blD14YpNinXedazX/1wQfvYvRtVZDoz+XmpPJGJ9hv/GRv2XN5gQwP65QUNbTIXGLpv"
+ + "HIebHywhGgLjxlzadiT6oqAzR6IPPrrco29vtiBJOtQNgAQ0X0yasf1kL79yUPrAE1slaSNU+iFGqq6LfFh6mT3IV7E+X3p8FYJghid6mjN6FX"
+ + "3+IJpaVUcb7uQQnpqOJbftM0ahukShilziCan6Vy9TY/l3rprRGaRwJwO8eroW+Tv5ENL6/VNWDrUG1fqh+9B9VGrderWvHx9hIuHakiK+roha"
+ + "4trPn83qvgTsWqj/O20yefMaSqnZaN4Y+ZqeZAHysBP9Op87FB0Aef+6+ClB/3NFAHsjuqjifWae2lYkR8jqOil7tS9RVk4tKtvQXYFv+RkxW7"
+ + "sTV0IUIjq709nf1z56Q1tVNu7wmw9DewDPONIwVkv7tOSQCiRpU3uCJramQwVeG2WVGj66LG77bReVOKwadrslh9y9pjl3m7/vS0y/d3nb09sp"
+ + "bbN0fftlLo8YPF+Qm/dxXGQtFJh7eM5VKa0HmvspjWZqOCCjW5QXzU6uCaxbnI0pnjJMedujoGhwPxm426UaAMyJJbsbYq6RdBZdv1s3QDn0l3"
+ + "bCmx6fXpaxZcN73tj1fPN1yHsJociK+T/Mez87PWWtb6xb4mYfntka1ltmbP2MJjHiW7eC8ogkQgT88IM1qwloBjzxPhGAHnYPPMETLV5/bBHc"
+ + "jpJf7bhPElwy+9ZSrmATfMvjeDYyfCM8medT478SJ3zxAnvNl3A/XDSKh6LHE0Cc3JPnHpZwShx5Hoy21VpSvNnJeNPRZ1VcFve6ZyA5azwaN8"
+ + "wFdjxxfGue2NlnFnk8Lk6h1YsdfPx4tDA/AXSBvNnVNVRRY4n83DDll9JNlZx4O23Gql+efLnsoBp0K6o8Aqd3+2QesVwEBcZvGya2PJtAzDiF"
+ + "erl8BDEbOpFKwrxd/8h6Qj/c/LpoLMkhet38GJWUWd0UBpwtzT/QRGLknHnNHFPOGgJdkg995d96D10RkNyIsQMU/frdos9WTxh9w43Sfc5RG5"
+ + "324Tz98CN5PEu02iZLMC+9Xb4GgIVhNz0wvJnlm004XaDwZjvVA5Wci/6z1IoZUD3yMtRDpj9UfW53JjimNF29n3njVTIHY13CeuUIljhkPRJu"
+ + "i9c7SWg9gyspBmtcb/3c9FyuLidyfbbkOcoocVA9QqMmtasYeN2Va4AajVqRnfVc9m+6ERg9SMgu0DtVnb3qCV5BFzWg/QZhduWOCKldp1LWhj"
+ + "Unqr2jwfsmC0nLXTAt9sQ8KM3dMbQZioAM1J3aP2evIVXWfd5ZaxHhGXg9RuN9fNMB5z1BhxWxkX3qHYWBEdZsYvDdF/Frdv5+Al6hyLeHyR8N"
+ + "Xujm1xlASBK+gYm9mxV5Nz4ZN15kLX05PusWE86zAz9emyRkptYbQN4u4cnbPQhBHnjCrCFgP0Net9pkl7MVSX40gs2mcF1XHsScr0GXOmheuQ"
+ + "afVPEuaT7GWk+zDUpx/mpsjsBGO3demNwVBB1beT4OOStczEh7/o8jgY4bUn/+meyjWxkOg773Co/jQ2GU3tbIoHUqApZ+1eBVwM4zl15DVd8i"
+ + "l3ybnqfhWqXDf7Y8L9zKGdJHnd5eYJqQF3ug/nAl1BI3EqZ+ZQMsvLhLqR7AYu+yt3U5XmeigNYaJ7IeixxFJUgLwB51shtp+DTEPjjAYKKSed"
+ + "Eew/k0/pho1N7FK/IbwmLRXqBCjZzrFuKetE6gzgdZ7h4AqRYdWG33XorjaDXEf/n+HCvEf+gq+p7vM04Nxnb/PttJMfqWAAz/V0oJ1mo3I99T"
+ + "10g+sDXiPmQsPczkUSa/hSp0h8ALLp8MEtAW5RI6n0APn2tIXkq/UGrvp7R5S79+7n5A9QBQHwCmmRFZPlWfsXNfrnQcWJ/Orapei1Fav3Q43y"
+ + "YREWafQy1udYoYcP+Av8uYwSZ7f+xm7Xh/AYz/rYefHx4mjx8NIjM1lSGWf3gY1RLhXqjfgKqG1rl9rD/q2DEaCnuvXxWmSZJOAbMfQa9benZV"
+ + "tv+ymriRpiy3/6nmHBcbFLm/n+r6nqoBwN+lOQ2A+zvFf8NkgCNBcMb/7p64Pzc7y18UNiUraFfPv4fCNfO5JNafhNy05hUM3D+fVAyZWgi3v0"
+ + "J5lTZKJ/bfkCpNslR6";
+
+ private static final String INT_TUPLE_620 =
+ "AgMJAQAKzJPoAwAAAAAAANNsNn32MENbAQAAAAAI8K3c5voSLQEAAAAAt79CeZU2SicBAAAAABXgItfNUycmAQAAAAAcwFoWl9IndwEAAAAAHX"
+ + "D3CaFBlCwBAAAAACOwqWFqcTVhAQAAAAAm0FrTBC2laQEAAAAAKCCEJYB8UQoBAAAAAIYvvbzF585oAQAAAAAwQLVd1sNmFQEAAAAAMRB0P69V"
+ + "+X0BAAAAADWgnWRvsuQiAQAAAAA5IEvBRWfFKgEAAAAAPZAlYAol9GEBAAAAAD7Asao+JXE8AQAAAABQ0CPMxtlqawEAAAAAVFDhYemXcjYBAA"
+ + "AAAFigfvAgpUMaAQAAAABaAJ5CODayLAEAAAAAW6BM3LSS50wBAAAAAFywfORWI9dWAQAAAABeECnhCObfFAEAAAAAaq/fmVxoaC0BAAAAAGBA"
+ + "No3dpERGAQAAAABn0PdgEwhMLgEAAAAA3J/8sdRHKG8BAAAAAHKw+D/L51p2AQAAAAB5sJhZ6mCzRAEAAAAACKDbpGhfrycBAAAAAH8QuX0Lix"
+ + "5FAQAAAACAYK91WgX6YwEAAAAAjNDJhkzJOiYBAAAAAJPATWK9NV1WAQAAAACakE9dp0rjcAEAAAAAo1AJ1GLPwxsBAAAAAKWQe+U0+W0DAQAA"
+ + "AACsUJqx6BUeIgEAAAAAB3A90e2WFi8BAAAAALnwFrkO/JhHAQAAAAC6QLPB2gZpXQEAAAAAwOD5osfFulMBAAAAAMQghgMWintPAQAAAADFkK"
+ + "5mVvV7HgEAAAAAySDeqCZ+vhkBAAAAAMpQJ9RO7j8+AQAAAADL4Cyew6kTVAEAAAAA1GBiKtOOsEwBAAAAANogcq9w7WYfAQAAAADbQA3s51DN"
+ + "BgEAAAAA3AB+582U/mQBAAAAAIxPrTHCHggkAQAAAADhkD2rkSVaMwEAAAAA47D+ZsLR6EEBAAAAAOSQ0uJvF9YMAQAAAADncCTkptsXFAEAAA"
+ + "AA9FDJp1rK2SEBAAAAAPWQU3wXTPk+AQAAAAD5UP1qwcWkNQEAAAAAACH0Xd2LFz8BAAAAAAERWtK/NUJxAQAAAAADISVUhMrYJgEAAAAABAG0"
+ + "QOTfylMBAAAAAA2RVMqG34AbAQAAAAATQZeDU7rFNQEAAAAAGtEwC5mMLyIBAAAAABsheG9wzD1MAQAAAAAiYYPxpzTteQEAAAAAKTFdNJ65gT"
+ + "cBAAAAADFxCaBzUk5lAQAAAAA54RYs4NL/LwEAAAAAOwH/obZD21YBAAAAAD3hFkwV5dE2AQAAAAA+objBjPwCMwEAAAAAQHH4FnGKJlQBAAAA"
+ + "AEPhdQc8Wp9qAQAAAABFMTSHfvgNNgEAAAAARsEO3QyCnkMBAAAAAEghSA8gtDJMAQAAAABNETqMK8/yCwEAAAAAUSF4d3eAZlgBAAAAAFVxDS"
+ + "KRF6ggAQAAAABjMZpXKy3mBwEAAAAAZiFB706BOFoBAAAAAHKwn0VEx5wPAQAAAABqkYsLC+csDwEAAAAAb7Fp6f2Gch0BAAAAAICgF1AyObQM"
+ + "AQAAAAB3ISX0/TEVPgEAAAAAe7GQCyhtxTQBAAAAAH9x93kO1zEzAQAAAACBQa96Okg9LQEAAAAAjYGflPkF/jwBAAAAAJDh3RQjyyYJAQAAAA"
+ + "CX0UW/4lsuPAEAAAAAmKEvKfUslysBAAAAAJqBoSX+O/lHAQAAAACisedzvXoLRwEAAAAAqMHaLV/6hwkBAAAAAKlB7ZfhP9URAQAAAACrIVUz"
+ + "Qv0paQEAAAAArcE6ZVC87kABAAAAALPBQbF2TXoaAQAAAAC2ocXHqEE5OwEAAAAAuCHkUO5rZg4BAAAAAMpwHyrrjiEWAQAAAAC7EayzlbvCGA"
+ + "EAAAAAvhEek4VG/ggBAAAAAMIxbI83A/ZeAQAAAADHYVj3QMRLUAEAAAAAyuEQukCGkjEBAAAAAMsxC4dlBgoUAQAAAADOEQl8MvkJJgEAAAAA"
+ + "0MFIMkvhTGMBAAAAANdxoeBrcl0yAQAAAADZYQ82iDQnGgEAAAAA4EFwTOKGOy0BAAAAAOUBdmFll1kfAQAAAADqITeGIdk+HgEAAAAA7XHqJr"
+ + "iFhDEBAAAAAPWhA2wR7sR/AQAAAADXIeEFbhnWEAEAAAAABOK4x5Pr5EoBAAAAAAZiJE6EyAY9AQAAAAAJAr1BsFikHQEAAAAACiLbtxyHXwEB"
+ + "AAAAAA/CJ8PXFUwrAQAAAAC5YO3N9cxdHAEAAAAAFLL6R7Zr+k8BAAAAABZSdtFhaOIFAQAAAAAX8gV5dkxXaQEAAAAAVTHet9NpnWgBAAAAAB"
+ + "2CduIgoUN0AQAAAAAfco3xbYEOCAEAAAAAIaLs19fWZ20BAAAAACmSB53AWW8FAQAAAADO4TLiQ3KvSQEAAAAALGInWNZ7FVIBAAAAAC9Cc+x2"
+ + "Ff9TAQAAAAAzAq4dA1rqeQEAAAAAIlG8idsmHSUBAAAAAEIifj99YB83AQAAAABG4sQyqyRVNQEAAAAARxJ/sZueq2wBAAAAAEliu5BitvwDAQ"
+ + "AAAABfII0E+BtIbQEAAAAAT/JN4LwMrhwBAAAAAFGyvHambw0fAQAAAABTUj9v2ImrFgEAAAAAVaLCUYj0JmoBAAAAAF6i9FF4VZJ7AQAAAABr"
+ + "AgyimGQzeAEAAAAAbMK9GajjG3kBAAAAAG3iqdZfxnZcAQAAAABwQqOPE6fGGwEAAAAAc2Jo3ktr/AMBAAAAAHRSx1E206tLAQAAAAB84iRSDh"
+ + "1MHAEAAAAAgMKxh2XazisBAAAAAFHBZMpJuNJ5AQAAAACIwkpQwuQXOwEAAAAAiQI8S09MaQUBAAAAAIuCPT4Fbc4NAQAAAABP4lF/X1vpTAEA"
+ + "AAAAjaK8GEqH7w8BAAAAAMoh6wMdmb0NAQAAAACYoqx9x4CQUAEAAAAAmRKxCIUTtzABAAAAAJvimc5OlGlaAQAAAACgEpTdH1K4VAEAAAAAqw"
+ + "Iimfe0viwBAAAAALrAxxVq7bExAQAAAACyYgui9ATVDgEAAAAAtHIbsHex138BAAAAALbyjolPX3IhAQAAAAC3YhopjQlLSAEAAAAAuGJ6Xm3A"
+ + "Zx8BAAAAAL0yc3JGkcwUAQAAAAC+srztyLjjWAEAAAAAwKKtlAgOWiIBAAAAAMHyiHh3vjVlAQAAAADD8rmNpV3yfQEAAAAA0HKMra+9snIBAA"
+ + "AAANGi/SxrFXU3AQAAAADT8jEaYsOnJwEAAAAA16JgJklv0DYBAAAAANlSpqR1R+J9AQAAAACUEhGwgLaIQwEAAAAA3zL7CD7MDUIBAAAAAOZi"
+ + "OG49f9MDAQAAAADoQiGg6SHGKAEAAAAA8FI6P1k4zVwBAAAAAPRCtpOvCcc9AQAAAAD2UiOBMwtXLwEAAAAAcIKMMZEqkF8BAAAAAAUDwCUQec"
+ + "IKAQAAAAAGE12faSmJHwEAAAAADsNBuTVnqDwBAAAAABCzcWVAAvZXAQAAAABwEg3Jq+1aPgEAAAAAFqMC1+5mPDwBAAAAABlDVKEtjHFpAQAA"
+ + "AAAbU+X7i4bUQQEAAAAAHYM8Ff6CaWoBAAAAACYDWJkL8HhNAQAAAAAoo3Ce33xfJAEAAAAA6AL6M7Yo+BQBAAAAACoDFntvuLYeAQAAAAAwY8"
+ + "cPj63aZgEAAAAAMcOHwOqcZ1cBAAAAADZTnZr9LMV8AQAAAACZcvITs05zUAEAAAAAOEOTUl0lSEoBAAAAAEwjz8SHea5VAQAAAABUs95zQ/JP"
+ + "JAEAAAAAWiNIaA67sW4BAAAAAFuT+Ugl2ocPAQAAAABiI7sh70mzPQEAAAAAZDN12p9Yvn8BAAAAAGcDiLbIMyhJAQAAAABt8/uIVY2WAgEAAA"
+ + "AAiyKPxPN9klsBAAAAALYiPjOjTlx5AQAAAACB01nnm/WPSgEAAAAAimPEo6QFpHABAAAAAIsz+InajQc9AQAAAACNA2HgOaHlWwEAAAAAkcP/"
+ + "jYuICVABAAAAAJKjvHD8lz8IAQAAAACVs7UKCBE8FwEAAAAAE7E2hhLfL3IBAAAAAJrDC6AKZGMvAQAAAACbo4QRBt3JGwEAAAAAnUP/4lWBoA"
+ + "QBAAAAAKDjWFFZQVY5AQAAAACms+9EGMRLNQEAAAAAtNN9Ojr1gWUBAAAAALgztRrDbnIDAQAAAAC8I7SEfvjrWwEAAAAAxBNQJtwVFxABAAAA"
+ + "AMajx89P9I4HAQAAAADJQ5tIB9LSJwEAAAAAygNhqucznxUBAAAAANbj+XRnLmdNAQAAAAApc3W4UHXhDAEAAAAA2XPFkMLPUHwBAAAAAOFTe4"
+ + "4md/g0AQAAAABwA7Tin1YsaQEAAAAA9OMLg1BZ2V8BAAAAAALEqdeD+3Z2AQAAAAAEZMYluDUJEgEAAAAABaRrw3IvuVcBAAAAAAdE3/vIz2Ax"
+ + "AQAAAAAKxHX0tvVkBQEAAAAACzRUObe56F8BAAAAAA8U6SPD4uI7AQAAAAATlPV/bsNKSAEAAAAAFBTCRQaBWm0BAAAAABVkEH+PS7ZCAQAAAA"
+ + "AZZIuMJ9iUEgEAAAAAG7TsqqhzYlQBAAAAAB4U+E1yjNRJAQAAAAAmpHUV9R2IXAEAAAAANXTFEwtIqEsBAAAAADv0c/yedyUMAQAAAABAtPt5"
+ + "ikBBOgEAAAAARtR7PD1qxVsBAAAAAEukQdA+seoeAQAAAABNxJv/8Hq1QQEAAAAAUIRYTnos0BcBAAAAAKZTbV6AZqkTAQAAAABU1PyeUxjnag"
+ + "EAAAAAWVRcfZVk1hkBAAAAAF20WTtBsxNXAQAAAABeRAcXahOiEQEAAAAAX5RaFY9jNWEBAAAAAGMUN6OcOtQyAQAAAABl1KBbsD3AfQEAAAAA"
+ + "aZSJUHYWk0sBAAAAAGpkuLRrNPZqAQAAAABttA/TgXIdSwEAAAAAbuTFyYTDSncBAAAAAHb0nshKEsssAQAAAACBhAkQyzruQAEAAAAAhFTvcu"
+ + "Ih40QBAAAAAIVkBxYGiSJ6AQAAAACK1GT9OzVMIgEAAAAAkNQMF+q0oTsBAAAAAJWUV6WnPrglAQAAAACYVDDZcx2mdgEAAAAAmtQYKDOmoS0B"
+ + "AAAAAJBUCEa4UyooAQAAAACdNFeOSWYtOgEAAAAAoMTR912wZyIBAAAAAKF0+aZnKWxxAQAAAACnlK8Co/n0YAEAAAAArQTEhbJx0RcBAAAAAL"
+ + "zDm1up0pkzAQAAAAC31BfXKcfffwEAAAAAuGS7VwygtWQBAAAAALm0+H132+phAQAAAAC6hJkumtpsRgEAAAAAwjTm4lmEHTsBAAAAAMYk1hJF"
+ + "/n4hAQAAAADHxIGFE39zAAEAAAAAyWT6gVpPNGMBAAAAAMwkIQqpXN0ZAQAAAADWwxcd1FLibgEAAAAA02SbCQaViXkBAAAAANSUHWf8ccxFAQ"
+ + "AAAADVxELbem9PbwEAAAAA18ToZMg9VBQBAAAAANy0IDTFJW1/AQAAAADd5IBItcImNAEAAAAA4PSL6pmDw3wBAAAAAOekucwfdmsdAQAAAADv"
+ + "9EA3BKhieAEAAAAA+OTb9Ap8WW0BAAAAAPp0s0wWsW4AAQAAAAD7BJZUHThQcAEAAAAA/bRhCXnaQlwBAAAAAP70PlkD+uN2AQAAAAAAZSzmRB"
+ + "vHcwEAAAAABBW5CKLNhHgBAAAAAAkFMT0B5E8zAQAAAAAMBSPTE8uNNwEAAAAADkU8ZQqTMGUBAAAAAKdkE2Wmj9cwAQAAAAAWNcsU0ZyKbgEA"
+ + "AAAAFyUQjG4v3lgBAAAAABilJsu7xV0JAQAAAAAaNYd8IgfoIwEAAAAAGxVZPD9Me20BAAAAABz11cVxBsQIAQAAAAAjpVs4Gv10HwEAAAAAMV"
+ + "VTWVFsvWoBAAAAADO1luENbEhQAQAAAABRRNdLFPmbWwEAAAAAPcXUpIEJgngBAAAAAD4VvA/H5okzAQAAAABCxdV12uiZewEAAAAARaVHM6ys"
+ + "3QQBAAAAAEnlI8xAu9gaAQAAAABNRak9Qi72EwEAAAAATgW2B0olym8BAAAAAE8lLAapo1YxAQAAAABR9TZNjffKRQEAAAAAUjWcO3Es/BsBAA"
+ + "AAAFnUVw3Q7/NPAQAAAABjxfJQV18YLAEAAAAAapW+KujbK1YBAAAAAGxFGtlWQMARAQAAAAButddTTU0NEAEAAAAAchU9X+A5YXsBAAAAAHhl"
+ + "c637iUsnAQAAAACDRX2imemuEQEAAAAAhqXf5udg1DMBAAAAAIeluvklMwl1AQAAAACIZVc30Qp0AQEAAAAAjpWNEGFd8AcBAAAAAJJ1BaZJ3t"
+ + "80AQAAAACU5ecxnYtgDwEAAAAAlvUejXjuPiQBAAAAAJxFlEVQCwk8AQAAAACj9fOXE4uSQAEAAAAAqXU6yOEwe1YBAAAAAKz1MEs7u/UUAQAA"
+ + "AAC0ZZoCCkf0CQEAAAAAt2Wl29aNjT8BAAAAALvVDXkjmuRSAQAAAAC9ZftTyybhKAEAAAAAwJUu8LU0yC4BAAAAAFmkVQ12gCR2AQAAAADGtU"
+ + "a2KroTOwEAAAAAyWWfUojJAz8BAAAAAMyFZE48wA1iAQAAAADPVVB60gaTEgEAAAAA1rXddOuwbF8BAAAAANsFu/45uVtCAQAAAADllTtknKho"
+ + "XwEAAAAA5yXfYohKEWABAAAAAOpFRgRmr81XAQAAAAD9hDfGqxbnfQEAAAAA8PV1MyfUGyUBAAAAAPOF0Rdl9sUIAQAAAAD5pTFJsvfhBwEAAA"
+ + "AA+VX4pXEl1C4BAAAAAAWGRWbqNOFcAQAAAAAQhgQk5e3AHQEAAAAAEfbwYciR43cBAAAAABi2gY/3XMxTAQAAAAAb1vNxPsH3XgEAAAAAJZZ5"
+ + "bIY+sgUBAAAAABHWaEb0OplXAQAAAAA0tvtcTuD+WwEAAAAANWZ/6Js3QFEBAAAAADfm0biwpAV5AQAAAAA8lnc5ILqAYAEAAAAAnBV9tF1JOj"
+ + "MBAAAAAEl20VxnpHRMAQAAAAAbdjFajlX/NQEAAAAATsb6GkfbrVABAAAAAE9G67Pr1SAVAQAAAABRVr/+meOHdgEAAAAAUkY+Iw/PgjABAAAA"
+ + "AFPmrrOH98JPAQAAAABaptYtBm6lXwEAAAAAXPa3Buw6YzoBAAAAAGJmfhc6avZiAQAAAABlRq+qE6RBIAEAAAAAZ1YkGQANSk8BAAAAAGm26d"
+ + "AcTCNvAQAAAABrxs43HYIREgEAAAAAcFbXUHVAnx0BAAAAAHOWbGhsGDYmAQAAAAB2xpAbu5ADEgEAAAAAhdYgu+zfwR8BAAAAAM/FtTlDANpB"
+ + "AQAAAACKJpy2IB+FbAEAAAAAjKZiWOmnzS4BAAAAAO71tLasEw5XAQAAAACOptONUTTlGwEAAAAAj2a9qUtT7BkBAAAAAJdWn4CLP0ROAQAAAA"
+ + "CZNt43R7I/KQEAAAAAmyY+ysxblwkBAAAAAGuGwcqQGrpoAQAAAACelkZHzdSDegEAAAAAn2YQQhnPJkUBAAAAAKG249GCBNwPAQAAAACkdjV9"
+ + "asaMWwEAAAAAp6ZfAO7FfUQBAAAAAKm2F0IpjgFIAQAAAACwNt4NUb11YQEAAAAAuQYDvUkBjU4BAAAAAMy0X1WfacY2AQAAAAC9xqO/IJfxTg"
+ + "EAAAAAv9aVWlxVH1cBAAAAAO7l5o4GB/1qAQAAAADPFrUBaFIOcgEAAAAA0bZysn0ZvCsBAAAAANXG+vIxZc07AQAAAADahnsJX0dFCAEAAAAA"
+ + "32Z/MIKCqRYBAAAAAKH2iWI9jwQGAQAAAADpppeAdpj4QgEAAAAA71Z8MD9XNiwBAAAAAPD2pOEVnX0pAQAAAAD3lvvVCUmqJQEAAAAA+ta4la"
+ + "/q+BUBAAAAAPvmQUkoKq1jAQAAAAD85hY2IplCTQEAAAAA/cbFSAae0QwBAAAAAP4WIRP7mLwQAQAAAAABRzmD4ZhTHwEAAAAAy9ZhQhLCwywB"
+ + "AAAAAAsXEPkydgAXAQAAAAAMJwWGHyl2JQEAAAAAD5dOMJJETj0BAAAAABOHmWDMP21fAQAAAAAZx8se+PMebwEAAAAAGhdfSWBfhzwBAAAAAF"
+ + "wGLshAd7lcAQAAAAArlw4eer9RYgEAAAAALbcQXk9zE0gBAAAAAA/3YH7RPcEPAQAAAAA7J2vj1fwtEAEAAAAARxcWLSFN8XoBAAAAAEjXBVUh"
+ + "/pMzAQAAAABOB9MYSH9vJwEAAAAAUOclDKUJLSIBAAAAAFFnAT7v4t0AAQAAAABSh541nnS9VQEAAAAAVKd2IM4aIVkBAAAAAFf3YKWmUDovAQ"
+ + "AAAABadx6o+96SXAEAAAAAW1d4zQwLrlsBAAAAAF9nG9hORA8TAQAAAABm19Ku5tdgQAEAAAAAaAf1vTlxWR4BAAAAAG0HbjeZbLQwAQAAAABv"
+ + "B9hVH2kBVwEAAAAAcFdWilz6EyIBAAAAAHKXKisEJX0qAQAAAABz94MfE9uHOQEAAAAA/Mb7B+KjxD0BAAAAAHnHhbexPndaAQAAAAB8V8+/pf"
+ + "D7aQEAAAAAUGf5zKgx4kIBAAAAAIL38arHUUxBAQAAAACJxwgkAAXFVgEAAAAAjqcOSJ8YWAUBAAAAAJGHb+THW65NAQAAAACSZ+fEPwIDVAEA"
+ + "AAAAmeekBw061W4BAAAAAKXnr2J5LeArAQAAAACqV/lIKGFKfQEAAAAArHcE7B5pJS8BAAAAAK1nsbOVWF8PAQAAAACy9+uDYUrLVgEAAAAAt6"
+ + "fHchNZ2xoBAAAAALpHtVRk4P4fAQAAAAC7d+hd9McPdgEAAAAAc0eibjlNmQIBAAAAAL23LRyDd1UbAQAAAADAVxmihtBjOwEAAAAAwceOZan7"
+ + "+kQBAAAAAMOX/BKBcJ0eAQAAAADEh+Q5oxC/RgEAAAAAxTedA+Pr7G0BAAAAAM4nmLhaz38VAQAAAADSt1UXVP3TXgEAAAAAcheLPeVuiwwBAA"
+ + "AAANrHRP3p+1I6AQAAAADblwTxetsiPgEAAAAA3Ucmp+NY/2QBAAAAANUnWDsuTVlUAQAAAADkx1gSmsQGAQEAAAAAkpeCNuW0+hQBAAAAAOin"
+ + "vWj+C4YEAQAAAADr53hbDf4ddgEAAAAA7vdlFbe3cGkBAAAAAPPnykeMx1duAQAAAAD3hwUoFutJEgEAAAAA+XfHDAk6ChABAAAAAAA4UdW/if"
+ + "k7AQAAAAASaKy5LGZHKwEAAAAAFTiTLx3TrjcBAAAAABjYp9MDriNkAQAAAAAfSCTacPgXIQEAAAAAIBjrqxSiszEBAAAAACEoVStvfwZzAQAA"
+ + "AAAjyN+m6JCcXwEAAAAAKhiLzkE7cR8BAAAAAC5ohvqvY00sAQAAAAA0CPX4kc+7QAEAAAAAS1jgely/vFwBAAAAAFTYRKC9bR5bAQAAAABmx6"
+ + "sYkegpIAEAAAAAWdhP4UJuPFcBAAAAAGBok4rzh34yAQAAAABhWBwM0oX9QgEAAAAAZBh4lmsLiFwBAAAAAGf4rfV42Ys/AQAAAABq2OPuMz1K"
+ + "LgEAAAAAb2jSF/bp+VIBAAAAAHIoREfE3E5eAQAAAAB82MXcLrpHdQEAAAAAfkjbthIfCmoBAAAAAIFYXmAgaidAAQAAAACEGJe647R1awEAAA"
+ + "AAhbijg24G3HABAAAAAHVXNA5TG+lRAQAAAACMGC/75fOEOgEAAAAAjXjZj2Cmgg8BAAAAANuXTffvn9B2AQAAAACamHH+5D9BIQEAAAAAFVjU"
+ + "Up5eO3kBAAAAAKT4ZsgNqF9RAQAAAACr6LqVsXRIIAEAAAAArCh5B8zWzwwBAAAAAK8Ige2n06ENAQAAAACzCExS8ZVIXQEAAAAAtogZZlASEF"
+ + "cBAAAAALiIzpEYshVvAQAAAAC9aPOwKqtsUgEAAAAAxXjBCjbv63IBAAAAAMbITN6IPSsjAQAAAADNaMqxN84NbQEAAAAA04goDpM+/CIBAAAA"
+ + "ANQoUNm08hoyAQAAAADXaG3e6o88NwEAAAAA5Ke3QJU7GX0BAAAAANqIQVuTsDNpAQAAAADhuOfIN0CjFgEAAAAA4ljTRiePtRkBAAAAAOg4wX"
+ + "hDGJkkAQAAAADpeF47IZnsQwEAAAAA6pgJOV03MH4BAAAAAAB4RzbgH58RAQAAAADwmCwi+ZO/KQEAAAAA8chgbP/T3FwBAAAAAPMod16MOB8T"
+ + "AQAAAACzGCJ64fp4LgEAAAAA+zh5iRMkjwEBAAAAAAGZQKijMXw3AQAAAAAHGbov/psjUgEAAAAAC1lgc0CTMAgBAAAAAAxZ0q4L5attAQAAAA"
+ + "CguCElBD3hNgEAAAAAxViylFsjKDEBAAAAABPpx2OzPp8VAQAAAAAV+X3LvYahBQEAAAAAIBneZbxjngsBAAAAAPCYAS+9blUxAQAAAAAnKb30"
+ + "SA7VHAEAAAAAKhkuTtL7vGwBAAAAADlJ4BjSUl5gAQAAAABNafXW1YsqJgEAAAAAcIcvm19DUj8BAAAAAFfpegoOI1l0AQAAAABYiW3BhIa5RA"
+ + "EAAAAAckhPhsXKJ1sBAAAAAGH5tQ4K4OUcAQAAAAABCQNKpKN8QgEAAAAA2liUxmNmsj8BAAAAAGkpZsfmPyN9AQAAAABqqcreJArtBQEAAAAA"
+ + "bMlNMFt8QHUBAAAAAG2pFrxKZmEeAQAAAAB0aZQGnF9lIQEAAAAAdZk2tVcy6BcBAAAAAIO5Msz5W38NAQAAAACHCQW29EGnNQEAAAAAkImrOX"
+ + "HAGBABAAAAAJNZ03drEH9IAQAAAACXKbshy5buKQEAAAAAq/iLKMymqRYBAAAAALDJ6KSB4BBRAQAAAACyiVvjz2O5KQEAAAAAtBkxgPRyJyUB"
+ + "AAAAAPvYvRtVZDozAQAAAAC5udU+LrjaWQEAAAAAWBnb5LjWHD0BAAAAAMuJKKs0Gtc5AQAAAADReU6ft2HHQgEAAAAA1blD14YpNikBAAAAAN"
+ + "YJ8jBO04VGAQAAAADXedazX/1wQQEAAAAAWBlk2QsPhCgBAAAAAOKp/uq7aaA8AQAAAADpibPEJykPKgEAAAAA76kW7oUJ51cBAAAAAMtpsXar"
+ + "m6RgAQAAAAD5eak8kYn2GwEAAAAA+mn5WTyigwgBAAAAAP3p5bvb/H1FAQAAAAD/GRv2XN5gQwEAAAAAA/rlBQ1tMhcBAAAAAAUKJ751ggYdAQ"
+ + "AAAAAMql7lifDuRgEAAAAAGLpvHIebHywBAAAAACEaAuPGXNp2AQAAAAAk+qKgM0eiDwEAAAAANQq1Gy5KfV0BAAAAAFcJN71SX7Z0AQAAAAA5"
+ + "yt5yMeV+RQEAAAAAPrrco29vtiABAAAAAJd5lV0SYfNiAQAAAABJqpj8ZYyVcQEAAAAATJqx/WQvv3IBAAAAAFD6wBNbJWkjAQAAAABSKsomdf"
+ + "1zKgEAAAAAVPohRqqui3wBAAAAAFaqpts5axQ0AQAAAABYepk9yFexPgEAAAAAX3p8FYJghicBAAAAAHqaM3oVff4gAQAAAAB+Cu/6YX1VJAEA"
+ + "AAAA1Ynt8rp0KgABAAAAAJpaVUcb7uQQAQAAAACemo4lt+0zRgEAAAAAobpEoYpc4gkBAAAAAKn6Vy9TY/l3AQAAAACumtEZpHAnAwEAAAAAsA"
+ + "oUO0XEWlQBAAAAALUKIyFOnykUAQAAAAC8eroW+Tv5EAEAAAAAsIrwgx+bE2oBAAAAAMgK07dUoyZeAQAAAABJOtQNgAQ0XwEAAAAA0vr9U1YO"
+ + "tQYBAAAAANUKUugeKMoTAQAAAADZSnjyOdmfXAEAAAAA3Xq1rx8fYSIBAAAAAN9qTDin/sQ+AQAAAADh2pIivq6IWgEAAAAA4trPn83qvgQBAA"
+ + "AAACQKPvxyxUAhAQAAAADBKlFkVBnqKAEAAAAA7Fqo/zttMnkBAAAAAN+KqPdgBhxRAQAAAADzKjeJDXRjVgEAAAAA/6lvDHvOfksBAAAAAPma"
+ + "nmQB8rATAQAAAAD7irflPvt7BgEAAAAA/TqfOxQdAHkBAAAAAP+6+ClB/3NFAQAAAAAAeyO6qOJ9ZgEAAAAA+UoomxIuCVIBAAAAAApLp8PJ42"
+ + "E5AQAAAADV+qH70H1UagEAAAAAIAujP4N/XksBAAAAACPr2aJeGB0pAQAAAAAl68JiLg6mJQEAAAAAJgttW2q2tnMBAAAAACl7tS9RVk4tAQAA"
+ + "AAAq29BdgW/5GQEAAAAALsvrWpgRgCYBAAAAACor2CD1YeIvAQAAAAAxW7sTV0IUIgEAAAAA8xpKqdlo3hgBAAAAADfrZzIbio4sAQAAAAA6u9"
+ + "PZ39c+egEAAAAAQksyUC9v/zEBAAAAAENbVTbu8JsPAQAAAABFCwRD02DAMAEAAAAAS/u05JAKJGkBAAAAAEN7AM840jBWAQAAAABTe4ImtqZD"
+ + "BQEAAAAAJYv5/7OrQC4BAAAAAFyrVEATPXk9AQAAAABeG2WVGj66LAEAAAAAaYu5kbhoVwgBAAAAAG77bReVOKwaAQAAAABvK7zg+WtdEQEAAA"
+ + "AAdrslh9y9pjkBAAAAAHebv+9LTL93AQAAAAB529PbKW2zdAEAAAAAeiurFapVXzcBAAAAAH37ZS6PGDxfAQAAAAB/CxYZInQ5WQEAAAAAiquI"
+ + "tvcdCR8BAAAAAI1rzffcLVRqAQAAAACQm/dxXGQtFAEAAAAAmHt4zlUprQcBAAAAAJr7KY1majggAQAAAACdi/lovrc2KwEAAAAAoOtrKoKwHj"
+ + "wBAAAAAKGLLmxnSyA1AQAAAACjW5QXzU6uCQEAAAAArFucjSmeMkwBAAAAAK7L0A3zAFYHAQAAAACxi+8tOzaZZAEAAAAAuNulGgDMiSUBAAAA"
+ + "ALrLaden8iIrAQAAAAC7G2KukXQWXQEAAAAAv1s3QDn0l3YBAAAAAMKbHp9elrFlAQAAAADDe97Y9XzzdQEAAAAAxasVMXcZYX0BAAAAAMh7Ca"
+ + "HIivk/AQAAAADMez87PWWtbwEAAAAA1ltmbP2MJjEBAAAAAOJbt4LyiCRCAQAAAADla2HuyIBEIAEAAAAA5gvHbkmvpQgBAAAAAOcrBHcmgiFG"
+ + "AQAAAADqS4qRYc5gIwEAAAAA7QtNbVK6BgUBAAAAAO8L0qsZ2f9lAQAAAAB526OgaHA/GQEAAAAArFviZh+e2RoBAAAAAPvr0BRSGQgdAQAAAA"
+ + "AE/PCDNasJaAEAAAAABjzxPhGAHnYBAAAAAA7s9HYnOg4YAQAAAAAPPMETLV5/bAEAAAAAEdyOkl/tuE8BAAAAABJcMvvWUq5gAQAAAAATfMvj"
+ + "eDYyfAEAAAAAG0zRSZWYUhUBAAAAAB9sBwgd6gBjAQAAAAAjPJnnU+O/EgEAAAAAJ3zxAnvNl3ABAAAAACvMP32+EYQrAQAAAAAyzKP8m0MqUw"
+ + "EAAAAAP1w0ioeixxMBAAAAAECc3JPnHpZwAQAAAABDbPDoFOA3CQEAAAAAShx5Hoy21VoBAAAAAFDsNSqp1UF9AQAAAABSvNnJeNPRZwEAAAAA"
+ + "VKyZSIwbOUIBAAAAAFVcFve6ZyA5AQAAAABfTMzVbyLoNAEAAAAAYgxqzQqSYkkBAAAAAGs8GjfMBXY8AQAAAABszH7eyFmmIwEAAAAAcMzp50"
+ + "etvXwBAAAAAHF8a57Y2WcWAQAAAAB5PC5OodWLHQEAAAAAfPx4tDA/AXQBAAAAAICM7OmbaN46AQAAAACBvNnVNVRRYwEAAAAAhUzoGGlj2XUB"
+ + "AAAAAIn83DDll9JNAQAAAABS7BlJHS1oBQEAAAAAlZx4O23Gql8BAAAAAJ58ueygGnQrAQAAAACfbC+l7sXMbwEAAAAArKtPBHtsw3kBAAAAAK"
+ + "YM04BTkpMbAQAAAACqPAKnd/tkHgEAAAAAsMzT74qUwXwBAAAAALFcBAXGbxsmAQAAAAC2PJtAzDiFegEAAAAAuXwEMRs6kUoBAAAAAI+MYYvQ"
+ + "0xEXAQAAAADCvF3/yHpCPwEAAAAAyax5wkmNKHYBAAAAAMqsqRfTLZsmAQAAAADTjD5cRmD8dwEAAAAA24znO+aMDjIBAAAAANz8umgsySF6AQ"
+ + "AAAADd/BiVlFndFAEAAAAA3qxr4eHZZQABAAAAAN+MwOZFR0sxAQAAAAB5PEu02iZLMAEAAAAA4ww1FskK7k4BAAAAAOScec0cU84aAQAAAADp"
+ + "7HdPqIB+FQEAAAAA8Oy/ILTErScBAAAAAPWs19W4/XsIAQAAAAD5rO0JvRgAVQEAAAAAAl2SD33l33oBAAAAAAPtHxW7YPg5AQAAAAAjDG9FwY"
+ + "KAZQEAAAAAlZyL/rPUihkBAAAAAAptFAkWZtURAQAAAAAPXRGQ3IixAwEAAAAAFP363aLPVk8BAAAAABeNiW2P3JInAQAAAAAYfcON0n3OUQEA"
+ + "AAAAG5324Tz98CMBAAAAABg9SMgu0DtVAQAAAAAiDXHyJYOWNQEAAAAAJc3zC1G/UG4BAAAAAC+9Xb4GgIVhAQAAAAA3PTC8meWbTQEAAAAAOF"
+ + "2g8GY71QMBAAAAALFspFHO3+1lAQAAAABQPfIy1EOmPwEAAAAAVI2SeZO2qEUBAAAAAJ/s7YhsgPwYAQAAAABdvZ9541UyBwEAAAAAY13CeuUI"
+ + "ljgBAAAAAGQ9Em6L1ztJAQAAAABmrWxYD5bQFQEAAAAAaD2DKykGa1wBAAAAAAacLc0/0ERiAQAAAABurQBnvkzoTgEAAAAAb/3c9FyuLicBAA"
+ + "AAAHJ9tuQ5yihxAQAAAAB43ZVrgBqNWgEAAAAAeu00T25WUjUBAAAAAH7NmNDe/b5aAQAAAACM7YRaoW7sFQEAAAAAkZ31XPZvuhEBAAAAAFA9"
+ + "QqMmtasYAQAAAACd7RswrqiyewEAAAAAoP0GYXbljggBAAAAAKENpQLmMGo3AQAAAACpXadS1oY1JwEAAAAAqr2jwfsmC0kBAAAAAMjtqwp7iD"
+ + "omAQAAAADLXTAt9sQ8KAEAAAAAzd0xtBmKgAwBAAAAANSd2j9nryFVAQAAAADWfd5ZaxHhGQEAAAAA2C0jd0u5NX4BAAAAAOftmOiTLspwAQAA"
+ + "AADqHYWBEdZsYgEAAAAA7A0lUKVPDWwBAAAAAAOtk/BdoXEUAQAAAADv7YyxdjvSKQEAAAAA8N0X8Wt2/n4BAAAAAJ296gleQRc1AQAAAACg7T"
+ + "qQUGqeVAEAAAAAXC09ifMEuEQBAAAAAAJeoci3h8kfAQAAAACe2lYkR8jqOgEAAAAADV7o5tcZQEgBAAAAAOc9QYcVsZF9AQAAAAAQ7k858xQj"
+ + "TgEAAAAAEr6Bib2bFXkBAAAAAByu8UORnWhzAQAAAABUfW53JjimNAEAAAAA1M06Yt81m0wBAAAAAAFuKTA9qGogAQAAAAAxDmv8EjoTQwEAAA"
+ + "AANz4ZN15kLX0BAAAAADk+6xYTzrMDAQAAAAA/XpskZKbWGwEAAAAAQN4u4cnbPQgBAAAAAEEeeMKsIWA/AQAAAABDXrfaZJezFQEAAAAARS50"
+ + "8Lo9jxMBAAAAAEdOIa0E2KA+AQAAAABJfjSCzaZwXQEAAAAATu7oUlLYCx0BAAAAAH5NUy4JtS9wAQAAAABXHsScr0GXOgEAAAAAXo6KOSRRAE"
+ + "MBAAAAAGFOzTVCf28LAQAAAABoXrkGn1TxLgEAAAAAaT7GWk+zDUoBAAAAAG0uiZ/x7UNfAQAAAABx/mpsjsBGOwEAAAAAd16Y3BUEHVsBAAAA"
+ + "AHk+DjkrXMxIAQAAAAB7/o8jgY4bUgEAAAAAf/6Z7KNbGQ4BAAAAAIO+9wqP40NhAQAAAAC2PC/cyhnSRwEAAAAAlN7WyKB1KgIBAAAAAJZ+1e"
+ + "BVwM4zAQAAAACXXkNV3yKXfAEAAAAAm56n4Vqlw38BAAAAAJ3eXmCakBd7AQAAAACg/nAl1BI3EgEAAAAApn5lAyy8uEsBAAAAAKkewGLvsrd1"
+ + "AQAAAACyHoscRSVICwEAAAAAt85zc3q/2AcBAAAAAMAedbIbafg0AQAAAADELlms8pshOQEAAAAAxD44wGCiknkBAAAAANEew/k0/phoAQAAAA"
+ + "DSDr/PgxqsaAEAAAAA1N7FK/IbwmIBAAAAANFeoEKNnOsWAQAAAADfTr/rOSE3JAEAAAAA4e4o4grYzUQBAAAAAGEORJL6tHtuAQAAAADRzpB7"
+ + "mVgmJQEAAAAAaS4kUMz7lB4BAAAAAO4eAKkWHVhtAQAAAAD3znnSg3/JBQEAAAAA+f4cK8R/6CoBAAAAAPqe7zNODcZ2AQAAAAD8jgFfXJAqBQ"
+ + "EAAAAA/z7bSTH6lgABAAAAAHg9RuN9fNMBAQAAAAAIb14ogYbqeQEAAAAADP9XSgnWajcBAAAAAA5vYxexqPJTAQAAAAASL9P0iRaMRwEAAAAA"
+ + "FI9xZS+ycXsBAAAAACPfU9dIPrA1AQAAAAA5XmeigNYaJwEAAAAAJr+FKnSHwAsBAAAAACdPaDfIBQkUAQAAAAAunwwS0BblEgEAAAAAOq80JU"
+ + "B8gRIBAAAAADxv2CTM5HYhAQAAAABAz8KxxqyMVwEAAAAA916K42g1xH8BAAAAAEq/UGrvp7R5AQAAAABLv37ufkD1AAEAAAAAOp9AD59rSF4B"
+ + "AAAAAFAfAKaZEVk+AQAAAABVn7FzX650HAEAAAAAWJ/Orapei1EBAAAAAFq/dDjfJhERAQAAAABeD4Lxu0JZMAEAAAAAZp9DLW51ihgBAAAAAO"
+ + "I+ZCw9zORRAQAAAABqj37gOnqmMwEAAAAAPI+JllT7vDoBAAAAAG+PqWEFHq0JAQAAAABw/4C/y5jBJgEAAAAA4p60TqDOB1kBAAAAAHmPvd1N"
+ + "/5BqAQAAAAB7f+xm7Xh/AQEAAAAAhq+CKKGzWWcBAAAAAIz/rYefHx4mAQAAAADR7rWxGbSTAgEAAAAAjx8NIjM1lSEBAAAAAFjPifjjvOM+AQ"
+ + "AAAACSbz+/xC5oXwEAAAAAln94GNUS4V4BAAAAAJ+vSXlU0ixzAQAAAAA8L3sYCg+7BwEAAAAApi+SCn0Xsm8BAAAAAKjfgKqG1rl9AQAAAACp"
+ + "D3gMUFB3CAEAAAAArD/q2DEaCnsBAAAAAK9fFaZJkk4BAQAAAACyb0VgFxA2DAEAAAAAsx9Br1t6dlUBAAAAALb/spq4kaYsAQAAAAC3/6nmHB"
+ + "cbFAEAAAAAub+f6vqeqgEBAAAAALzvMl6Kh6xMAQAAAAC/7wCNc4wwfgEAAAAAwN+lOQ2A+zsBAAAAAMV/w2SAI0FwAQAAAADG/+6euD83OwEA"
+ + "AAAAy18UNiUraFcBAAAAAM+/h8I187kkAQAAAADWn4TctOYVDAEAAAAA2u/BfxdT1FEBAAAAANw/n1QMmVoIAQAAAADnT3l5YHL0fwEAAAAA6K"
+ + "/8/dhvlREBAAAAAO5PvTdDwu9yAQAAAADvr67PW930AQEAAAAA9t+QKk2yVHoBAAAAAP6PSfiLQq5UAQAAAAA=";
+
+ private static final String KLL_DOUBLES_620 =
+ "BQEPAMgACADoAwAAAAAAAMgAAwBiAAAAxAAAAMUAAAAAAAAAAAAAAAAAAAAAOI9AAAAAAAA4j0AAAAAAADCPQAAAAAAAKI9AAAAAAAAgj0AAAA"
+ + "AAABiPQAAAAAAAEI9AAAAAAAAIj0AAAAAAAACPQAAAAAAA+I5AAAAAAADwjkAAAAAAAOiOQAAAAAAA4I5AAAAAAADYjkAAAAAAANCOQAAAAAAA"
+ + "yI5AAAAAAADAjkAAAAAAALiOQAAAAAAAsI5AAAAAAACojkAAAAAAAKCOQAAAAAAAmI5AAAAAAACQjkAAAAAAAIiOQAAAAAAAgI5AAAAAAAB4jk"
+ + "AAAAAAAHCOQAAAAAAAaI5AAAAAAABgjkAAAAAAAFiOQAAAAAAAUI5AAAAAAABIjkAAAAAAAECOQAAAAAAAOI5AAAAAAAAwjkAAAAAAACiOQAAA"
+ + "AAAAII5AAAAAAAAYjkAAAAAAABCOQAAAAAAACI5AAAAAAAAAjkAAAAAAAPiNQAAAAAAA8I1AAAAAAADojUAAAAAAAOCNQAAAAAAA2I1AAAAAAA"
+ + "DQjUAAAAAAAMiNQAAAAAAAwI1AAAAAAAC4jUAAAAAAALCNQAAAAAAAqI1AAAAAAACgjUAAAAAAAJiNQAAAAAAAkI1AAAAAAACIjUAAAAAAAICN"
+ + "QAAAAAAAeI1AAAAAAABwjUAAAAAAAGiNQAAAAAAAYI1AAAAAAABYjUAAAAAAAFCNQAAAAAAASI1AAAAAAABAjUAAAAAAADiNQAAAAAAAMI1AAA"
+ + "AAAAAojUAAAAAAACCNQAAAAAAAGI1AAAAAAAAQjUAAAAAAAAiNQAAAAAAAAI1AAAAAAAD4jEAAAAAAAPCMQAAAAAAA6IxAAAAAAADgjEAAAAAA"
+ + "ANiMQAAAAAAA0IxAAAAAAADIjEAAAAAAAMCMQAAAAAAAuIxAAAAAAACwjEAAAAAAAKiMQAAAAAAAoIxAAAAAAACYjEAAAAAAAJCMQAAAAAAAiI"
+ + "xAAAAAAACAjEAAAAAAAHiMQAAAAAAAcIxAAAAAAABojEAAAAAAAGCMQAAAAAAAWIxAAAAAAABQjEAAAAAAAEiMQAAAAAAAQIxAAAAAAAA4jEAA"
+ + "AAAAADCMQAAAAAAAAHtAAAAAAAAACEAAAAAAAAAcQAAAAAAAACZAAAAAAAAALkAAAAAAAAAzQAAAAAAAADdAAAAAAAAAO0AAAAAAAAA/QAAAAA"
+ + "AAgEFAAAAAAACAQ0AAAAAAAIBFQAAAAAAAgEdAAAAAAACASUAAAAAAAIBLQAAAAAAAgE1AAAAAAACAT0AAAAAAAMBQQAAAAAAAwFFAAAAAAADA"
+ + "UkAAAAAAAMBTQAAAAAAAwFRAAAAAAADAVUAAAAAAAMBWQAAAAAAAwFdAAAAAAADAWEAAAAAAAMBZQAAAAAAAwFpAAAAAAADAW0AAAAAAAMBcQA"
+ + "AAAAAAwF1AAAAAAADAXkAAAAAAAMBfQAAAAAAAYGBAAAAAAADgYEAAAAAAAGBhQAAAAAAA4GFAAAAAAABgYkAAAAAAAOBiQAAAAAAAYGNAAAAA"
+ + "AADgY0AAAAAAAGBkQAAAAAAA4GRAAAAAAABgZUAAAAAAAOBlQAAAAAAAYGZAAAAAAADgZkAAAAAAAGBnQAAAAAAA4GdAAAAAAABgaEAAAAAAAO"
+ + "BoQAAAAAAAYGlAAAAAAADgaUAAAAAAAGBqQAAAAAAA4GpAAAAAAABga0AAAAAAAOBrQAAAAAAAYGxAAAAAAADgbEAAAAAAAGBtQAAAAAAA4G1A"
+ + "AAAAAABgbkAAAAAAAOBuQAAAAAAAYG9AAAAAAADgb0AAAAAAADBwQAAAAAAAcHBAAAAAAACwcEAAAAAAAPBwQAAAAAAAMHFAAAAAAABwcUAAAA"
+ + "AAALBxQAAAAAAA8HFAAAAAAAAwckAAAAAAAHByQAAAAAAAsHJAAAAAAADwckAAAAAAADBzQAAAAAAAcHNAAAAAAACwc0AAAAAAAPBzQAAAAAAA"
+ + "MHRAAAAAAABwdEAAAAAAALB0QAAAAAAA8HRAAAAAAAAwdUAAAAAAAHB1QAAAAAAAsHVAAAAAAADwdUAAAAAAADB2QAAAAAAAcHZAAAAAAACwdk"
+ + "AAAAAAAPB2QAAAAAAAMHdAAAAAAABwd0AAAAAAALB3QAAAAAAA8HdAAAAAAAAweEAAAAAAAHB4QAAAAAAAsHhAAAAAAADweEAAAAAAADB5QAAA"
+ + "AAAAcHlAAAAAAACweUAAAAAAAPB5QAAAAAAAMHpAAAAAAABwekAAAAAAALB6QAAAAAAA8HpAAAAAAAAge0AAAAAAAGB7QAAAAAAAoHtAAAAAAA"
+ + "Dge0AAAAAAACB8QAAAAAAAYHxAAAAAAACgfEAAAAAAAOB8QAAAAAAAIH1AAAAAAABgfUAAAAAAAKB9QAAAAAAA4H1AAAAAAAAgfkAAAAAAAGB+"
+ + "QAAAAAAAoH5AAAAAAADgfkAAAAAAACB/QAAAAAAAYH9AAAAAAACgf0AAAAAAAOB/QAAAAAAAEIBAAAAAAAAwgEAAAAAAAFCAQAAAAAAAcIBAAA"
+ + "AAAACQgEAAAAAAALCAQAAAAAAA0IBAAAAAAADwgEAAAAAAABCBQAAAAAAAMIFAAAAAAABQgUAAAAAAAHCBQAAAAAAAkIFAAAAAAACwgUAAAAAA"
+ + "ANCBQAAAAAAA8IFAAAAAAAAQgkAAAAAAADCCQAAAAAAAUIJAAAAAAABwgkAAAAAAAJCCQAAAAAAAsIJAAAAAAADQgkAAAAAAAPCCQAAAAAAAEI"
+ + "NAAAAAAAAwg0AAAAAAAFCDQAAAAAAAcINAAAAAAACQg0AAAAAAALCDQAAAAAAA0INAAAAAAADwg0AAAAAAABCEQAAAAAAAMIRAAAAAAABQhEAA"
+ + "AAAAAHCEQAAAAAAAkIRAAAAAAACwhEAAAAAAANCEQAAAAAAA8IRAAAAAAAAQhUAAAAAAADCFQAAAAAAAUIVAAAAAAABwhUAAAAAAAJCFQAAAAA"
+ + "AAsIVAAAAAAADQhUAAAAAAAPCFQAAAAAAAEIZAAAAAAAAwhkAAAAAAAFCGQAAAAAAAcIZAAAAAAACQhkAAAAAAALCGQAAAAAAA0IZAAAAAAADw"
+ + "hkAAAAAAABCHQAAAAAAAMIdAAAAAAABYh0AAAAAAAHiHQAAAAAAAmIdAAAAAAAC4h0AAAAAAANiHQAAAAAAA+IdAAAAAAAAYiEAAAAAAADiIQA"
+ + "AAAAAAWIhAAAAAAAB4iEAAAAAAAJiIQAAAAAAAuIhAAAAAAADYiEAAAAAAAPiIQAAAAAAAGIlAAAAAAAA4iUAAAAAAAFiJQAAAAAAAeIlAAAAA"
+ + "AACYiUAAAAAAALiJQAAAAAAA2IlAAAAAAAD4iUAAAAAAABiKQAAAAAAAOIpAAAAAAABYikAAAAAAAHiKQAAAAAAAmIpAAAAAAAC4ikAAAAAAAN"
+ + "iKQAAAAAAA+IpAAAAAAAAYi0AAAAAAADiLQAAAAAAAWItAAAAAAAB4i0AAAAAAAJiLQAAAAAAAuItAAAAAAADYi0AAAAAAAPiLQAAAAAA"
+ + "AGIxA";
+
+ private static final String CPC_620 =
+ "CAEQDAAOzJO/AwAAqQAAAAAAAIDpf61AukbLwgc7j0DFwrQAo3IWU1I/O7Fs5pj30w05ooCjuSCv+qlPTMUZ/YSP8XJoRlRtm1Zo0zi8QGbquA"
+ + "tDJJ2ZqXwsxUyanrnuEF8YFbMrKnd94+Z2iBmrn0ZcUJXEZ8dqxQsbUzSxMbaGgghJ9D1/12xUeOPmRQ3UOeslUYJeO/Y8ITE8F8pt2RgIRCd9"
+ + "SlxJop/nCMqKymds6z1fum0e+mUU0anDeD0nyKtariSL39evohMifJRRjHAUixgxml109au5sbgCZkrF6PUQJyy8JpbAIire8rvDv+wZKbMTNT"
+ + "MdqzdSraPB1F6z8j6TnLSusYRjrGhG5Ykniio2cofwhKRTYvbESpWWE8f8e+QDUzk5RMP8IbpNYv25Y3q1Q5SIU0KCiRKjb0rnvWiysfYp+dFa"
+ + "3HxOOTM1mlVVTVQz8PYMxGm78tRR+NC+NTVlO1VJVi2mkSdbWhHJdCz0aXxEGz1Fqp+a6VeyF+XGvd2YJ8iqq9YnFZ29Ue5fmWkZ0Q/fGrYYJf"
+ + "SmzXRUJZ3Thr6bmsIF025KYGb7jrXbzUydOZvhd+x82xsrldy1tfHvZaSiRC0zM4g+GzwhiEctxmJ8V5WmiYVXiNksedLJldp4K0lejtLqyE53"
+ + "P4PIrc1ME4hsSRA+Z3T8RD9mSjbU7U6BkCYsI6VpCilgjcP58mY1Tm0FCwh1N/KSUrWEkKRFl401PTLvnzn/RNmsqZRMnO6EVmfIeYJm79dxV3"
+ + "PSJtrcNjqx7BOo4/5IckW2wxAiVowpGhOmBRqtNtQWX9vaWqvuw2kzcrulaqM1OFOMVOBStTTkC7UnzpKLWSWaWSMbfQ+zxNvUm0nrzZpH2r7j"
+ + "SDujrWui5UgKCdNd94kmdLmyuhBVf2ObtYnqdZw2eq68mdoGvVFdGG3rI7RlGAAA";
+
+ private static final String FREQ_LONGS_620 =
+ "BAEKCAQAAAAKAAAAAAAAAOgDAAAAAAAAAAAAAAAAAABkAAAAAAAAAGQAAAAAAAAAZAAAAAAAAABkAAAAAAAAAGQAAAAAAAAAZAAAAAAAAABkAA"
+ + "AAAAAAAGQAAAAAAAAAZAAAAAAAAABkAAAAAAAAAAAAAAAAAAAABgAAAAAAAAAFAAAAAAAAAAQAAAAAAAAAAgAAAAAAAAAIAAAAAAAAAAkAAAAA"
+ + "AAAAAQAAAAAAAAAHAAAAAAAAAAMAAAAAAAAA";
+
+ private static final String FREQ_STRINGS_620 =
+ "BAEKCAQAAAAKAAAAAAAAAOgDAAAAAAAAAAAAAAAAAABkAAAAAAAAAGQAAAAAAAAAZAAAAAAAAABkAAAAAAAAAGQAAAAAAAAAZAAAAAAAAABkAA"
+ + "AAAAAAAGQAAAAAAAAAZAAAAAAAAABkAAAAAAAAAAIAAABzOAIAAABzNgIAAABzNAIAAABzMAIAAABzMgIAAABzNQIAAABzNwIAAABzOQIAAABz"
+ + "MQIAAABzMw==";
+
+ private static byte[] decode(String base64) {
+ return Base64.getDecoder().decode(base64);
+ }
+
+ /// Wraps the fixture in a ByteBuffer with a non-zero position to mimic the query-path
+ /// `deserializeIntermediateResult` flow reading from a sliced DataTable buffer.
+ private static ByteBuffer offsetBuffer(byte[] bytes) {
+ ByteBuffer byteBuffer = ByteBuffer.allocate(bytes.length + 5);
+ byteBuffer.position(5);
+ byteBuffer.put(bytes);
+ byteBuffer.position(5);
+ return byteBuffer;
+ }
+
+ @Test
+ public void testThetaSketch620() {
+ for (String fixture : new String[]{THETA_ORDERED_620, THETA_UNORDERED_620}) {
+ byte[] bytes = decode(fixture);
+ assertEquals(ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(bytes).getEstimate(), 1000.0);
+ assertEquals(ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(ByteBuffer.wrap(bytes)).getEstimate(),
+ 1000.0);
+ assertEquals(ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(offsetBuffer(bytes)).getEstimate(), 1000.0);
+ assertEquals(CustomSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(bytes).getEstimate(), 1000.0);
+ }
+ }
+
+ @Test
+ public void testIntegerTupleSketch620() {
+ byte[] bytes = decode(INT_TUPLE_620);
+ TupleSketch<IntegerSummary> sketch = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(bytes);
+ assertEquals(sketch.getEstimate(), 1000.0);
+ assertEquals(ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(offsetBuffer(bytes)).getEstimate(),
+ 1000.0);
+ assertEquals(CustomSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(bytes).getEstimate(), 1000.0);
+ }
+
+ @Test
+ public void testKllDoublesSketch620() {
+ byte[] bytes = decode(KLL_DOUBLES_620);
+ KllDoublesSketch sketch = ObjectSerDeUtils.KLL_SKETCH_SER_DE.deserialize(bytes);
+ assertEquals(sketch.getN(), 1000);
+ assertEquals(sketch.getMinItem(), 0.0);
+ assertEquals(sketch.getMaxItem(), 999.0);
+ assertEquals(ObjectSerDeUtils.KLL_SKETCH_SER_DE.deserialize(offsetBuffer(bytes)).getN(), 1000);
+ }
+
+ @Test
+ public void testCpcSketch620() {
+ byte[] bytes = decode(CPC_620);
+ // Estimate computed by datasketches-java 6.2.0 from this exact serialized image; the ICON estimator is
+ // deterministic, so the value must be stable across library versions.
+ CpcSketch sketch = ObjectSerDeUtils.DATA_SKETCH_CPC_SER_DE.deserialize(bytes);
+ assertEquals(sketch.getEstimate(), 999.378789508911, 1.0);
+ assertEquals(ObjectSerDeUtils.DATA_SKETCH_CPC_SER_DE.deserialize(offsetBuffer(bytes)).getEstimate(),
+ 999.378789508911, 1.0);
+ assertEquals(CustomSerDeUtils.DATA_SKETCH_CPC_SER_DE.deserialize(bytes).getEstimate(), 999.378789508911, 1.0);
+ }
+
+ @Test
+ public void testFrequentLongsSketch620() {
+ byte[] bytes = decode(FREQ_LONGS_620);
+ FrequentLongsSketch sketch = ObjectSerDeUtils.FREQUENT_LONGS_SKETCH_SER_DE.deserialize(bytes);
+ for (long item = 0; item < 10; item++) {
+ assertEquals(sketch.getEstimate(item), 100);
+ }
+ assertEquals(ObjectSerDeUtils.FREQUENT_LONGS_SKETCH_SER_DE.deserialize(offsetBuffer(bytes)).getEstimate(0L), 100);
+ }
+
+ @Test
+ public void testFrequentStringsSketch620() {
+ byte[] bytes = decode(FREQ_STRINGS_620);
+ FrequentItemsSketch<String> sketch = ObjectSerDeUtils.FREQUENT_STRINGS_SKETCH_SER_DE.deserialize(bytes);
+ for (int item = 0; item < 10; item++) {
+ assertEquals(sketch.getEstimate("s" + item), 100);
+ }
+ assertEquals(ObjectSerDeUtils.FREQUENT_STRINGS_SKETCH_SER_DE.deserialize(offsetBuffer(bytes)).getEstimate("s0"),
+ 100);
+ }
+}
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/common/ObjectSerDeUtilsTest.java b/pinot-core/src/test/java/org/apache/pinot/core/common/ObjectSerDeUtilsTest.java
index 5dd7c023ab8..cb90f7f4245 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/common/ObjectSerDeUtilsTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/common/ObjectSerDeUtilsTest.java
@@ -44,13 +44,13 @@ import java.util.PriorityQueue;
import java.util.Random;
import org.apache.commons.lang3.RandomStringUtils;
import org.apache.datasketches.cpc.CpcSketch;
-import org.apache.datasketches.theta.SetOperationBuilder;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.Sketches;
-import org.apache.datasketches.theta.UpdateSketch;
-import org.apache.datasketches.tuple.aninteger.IntegerSketch;
+import org.apache.datasketches.theta.ThetaSetOperationBuilder;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
+import org.apache.datasketches.tuple.TupleSketch;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.datasketches.tuple.aninteger.IntegerSummarySetOperations;
+import org.apache.datasketches.tuple.aninteger.IntegerTupleSketch;
import org.apache.pinot.core.query.aggregation.function.PercentileEstAggregationFunction;
import org.apache.pinot.core.query.aggregation.function.PercentileTDigestAggregationFunction;
import org.apache.pinot.core.query.aggregation.function.funnel.FunnelStepEvent;
@@ -549,7 +549,7 @@ public class ObjectSerDeUtilsTest {
@Test
public void testThetaSketch() {
for (int i = 0; i < NUM_ITERATIONS; i++) {
- UpdateSketch input = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input = UpdatableThetaSketch.builder().build();
int size = RANDOM.nextInt(100) + 10;
boolean shouldOrder = RANDOM.nextBoolean();
@@ -557,10 +557,10 @@ public class ObjectSerDeUtilsTest {
input.update(j);
}
- Sketch sketch = input.compact(shouldOrder, null);
+ ThetaSketch sketch = input.compact(shouldOrder, null);
byte[] bytes = ObjectSerDeUtils.serialize(sketch);
- Sketch actual = ObjectSerDeUtils.deserialize(bytes, ObjectSerDeUtils.ObjectType.DataSketch);
+ ThetaSketch actual = ObjectSerDeUtils.deserialize(bytes, ObjectSerDeUtils.ObjectType.DataSketch);
assertEquals(actual.getEstimate(), sketch.getEstimate(), ERROR_MESSAGE);
assertEquals(actual.toByteArray(), sketch.toByteArray(), ERROR_MESSAGE);
@@ -571,16 +571,16 @@ public class ObjectSerDeUtilsTest {
@Test
public void testThetaSketchAccumulator() {
for (int i = 0; i < NUM_ITERATIONS; i++) {
- UpdateSketch input = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input = UpdatableThetaSketch.builder().build();
int size = RANDOM.nextInt(100) + 10;
for (int j = 0; j < size; j++) {
input.update(j);
}
- SetOperationBuilder setOperationBuilder = new SetOperationBuilder();
+ ThetaSetOperationBuilder setOperationBuilder = new ThetaSetOperationBuilder();
ThetaSketchAccumulator accumulator = new ThetaSketchAccumulator(setOperationBuilder, 2);
- Sketch sketch = input.compact(false, null);
+ ThetaSketch sketch = input.compact(false, null);
accumulator.apply(sketch);
byte[] bytes = ObjectSerDeUtils.serialize(accumulator);
@@ -597,7 +597,7 @@ public class ObjectSerDeUtilsTest {
for (int i = 0; i < NUM_ITERATIONS; i++) {
int lgK = 4;
int size = RANDOM.nextInt(100) + 10;
- IntegerSketch input = new IntegerSketch(lgK, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch input = new IntegerTupleSketch(lgK, IntegerSummary.Mode.Sum);
for (int j = 0; j < size; j++) {
input.update(j, RANDOM.nextInt(100));
@@ -606,7 +606,7 @@ public class ObjectSerDeUtilsTest {
IntegerSummarySetOperations setOps =
new IntegerSummarySetOperations(IntegerSummary.Mode.Sum, IntegerSummary.Mode.Sum);
TupleIntSketchAccumulator accumulator = new TupleIntSketchAccumulator(setOps, (int) Math.pow(2, lgK), 2);
- org.apache.datasketches.tuple.Sketch<IntegerSummary> sketch = input.compact();
+ TupleSketch<IntegerSummary> sketch = input.compact();
accumulator.apply(sketch);
byte[] bytes = ObjectSerDeUtils.serialize(accumulator);
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/function/scalar/SketchFunctionsTest.java b/pinot-core/src/test/java/org/apache/pinot/core/function/scalar/SketchFunctionsTest.java
index 11662741db9..a0f66b1e5fb 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/function/scalar/SketchFunctionsTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/function/scalar/SketchFunctionsTest.java
@@ -19,11 +19,10 @@
package org.apache.pinot.core.function.scalar;
import com.dynatrace.hash4j.distinctcount.UltraLogLog;
+import java.lang.foreign.MemorySegment;
import java.math.BigDecimal;
import org.apache.datasketches.cpc.CpcSketch;
-import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.Sketches;
+import org.apache.datasketches.theta.ThetaSketch;
import org.apache.pinot.core.common.ObjectSerDeUtils;
import org.apache.pinot.segment.local.utils.UltraLogLogUtils;
import org.testng.Assert;
@@ -57,7 +56,7 @@ public class SketchFunctionsTest {
@Test
public void thetaThetaSketchSummary() {
for (Object i : _inputs) {
- Sketch sketch = Sketches.wrapSketch(Memory.wrap(SketchFunctions.toThetaSketch(i)));
+ ThetaSketch sketch = ThetaSketch.wrap(MemorySegment.ofArray(SketchFunctions.toThetaSketch(i)).asReadOnly());
Assert.assertEquals(SketchFunctions.thetaSketchToString(sketch), sketch.toString());
}
Assert.assertThrows(RuntimeException.class, () -> SketchFunctions.thetaSketchToString(new Object()));
@@ -113,7 +112,7 @@ public class SketchFunctionsTest {
@Test
public void thetaCpcSketchToString() {
for (Object i : _inputs) {
- CpcSketch sketch = CpcSketch.heapify(Memory.wrap(SketchFunctions.toCpcSketch(i)));
+ CpcSketch sketch = CpcSketch.heapify(MemorySegment.ofArray(SketchFunctions.toCpcSketch(i)));
Assert.assertEquals(SketchFunctions.cpcSketchToString(sketch), sketch.toString());
}
Assert.assertThrows(RuntimeException.class, () -> SketchFunctions.cpcSketchToString(new Object()));
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/DistinctCountCPCSketchAggregationFunctionTest.java b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/DistinctCountCPCSketchAggregationFunctionTest.java
index 17207ab3b45..a3bcda6204d 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/DistinctCountCPCSketchAggregationFunctionTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/query/aggregation/function/DistinctCountCPCSketchAggregationFunctionTest.java
@@ -18,11 +18,11 @@
*/
package org.apache.pinot.core.query.aggregation.function;
+import java.lang.foreign.MemorySegment;
import java.util.Base64;
import java.util.List;
import java.util.Map;
import org.apache.datasketches.cpc.CpcSketch;
-import org.apache.datasketches.memory.Memory;
import org.apache.pinot.common.request.Literal;
import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.common.utils.DataSchema.ColumnDataType;
@@ -110,7 +110,7 @@ public class DistinctCountCPCSketchAggregationFunctionTest {
// The string should be a valid Base64-encoded CPC sketch that round-trips
String base64 = (String) converted;
byte[] bytes = Base64.getDecoder().decode(base64);
- CpcSketch deserialized = CpcSketch.heapify(Memory.wrap(bytes));
+ CpcSketch deserialized = CpcSketch.heapify(MemorySegment.ofArray(bytes));
Assert.assertEquals(deserialized.getEstimate(), 0.0);
}
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/aggregator/DistinctCountThetaSketchAggregatorTest.java b/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/aggregator/DistinctCountThetaSketchAggregatorTest.java
index 0c416762e2b..62ef1de6b47 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/aggregator/DistinctCountThetaSketchAggregatorTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/aggregator/DistinctCountThetaSketchAggregatorTest.java
@@ -21,8 +21,8 @@ package org.apache.pinot.core.segment.processing.aggregator;
import java.util.HashMap;
import java.util.Map;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.UpdateSketch;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
import org.apache.pinot.core.common.ObjectSerDeUtils;
import org.apache.pinot.segment.spi.Constants;
import org.testng.annotations.BeforeMethod;
@@ -41,23 +41,23 @@ public class DistinctCountThetaSketchAggregatorTest {
@Test
public void testAggregateWithDefaultBehaviour() {
- Sketch firstSketch = createThetaSketch(64);
- Sketch secondSketch = createThetaSketch(32);
+ ThetaSketch firstSketch = createThetaSketch(64);
+ ThetaSketch secondSketch = createThetaSketch(32);
byte[] value1 = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(firstSketch);
byte[] value2 = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(secondSketch);
Map<String, String> functionParameters = new HashMap<>();
byte[] result = (byte[]) _thetaSketchAggregator.aggregate(value1, value2, functionParameters);
- Sketch resultSketch = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(result);
+ ThetaSketch resultSketch = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(result);
assertNotNull(resultSketch);
assertEquals(resultSketch.getRetainedEntries(), 64);
}
@Test
public void testAggregateWithNominalEntries() {
- Sketch firstSketch = createThetaSketch(64);
- Sketch secondSketch = createThetaSketch(32);
+ ThetaSketch firstSketch = createThetaSketch(64);
+ ThetaSketch secondSketch = createThetaSketch(32);
byte[] value1 = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(firstSketch);
byte[] value2 = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(secondSketch);
@@ -66,15 +66,15 @@ public class DistinctCountThetaSketchAggregatorTest {
byte[] result = (byte[]) _thetaSketchAggregator.aggregate(value1, value2, functionParameters);
- Sketch resultSketch = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(result);
+ ThetaSketch resultSketch = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(result);
assertNotNull(resultSketch);
assertEquals(resultSketch.getRetainedEntries(), 32);
}
@Test
public void testAggregateWithSamplingProbability() {
- Sketch firstSketch = createThetaSketch(64);
- Sketch secondSketch = createThetaSketch(32);
+ ThetaSketch firstSketch = createThetaSketch(64);
+ ThetaSketch secondSketch = createThetaSketch(32);
byte[] value1 = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(firstSketch);
byte[] value2 = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(secondSketch);
@@ -83,13 +83,13 @@ public class DistinctCountThetaSketchAggregatorTest {
byte[] result = (byte[]) _thetaSketchAggregator.aggregate(value1, value2, functionParameters);
- Sketch resultSketch = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(result);
+ ThetaSketch resultSketch = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(result);
assertNotNull(resultSketch);
assertTrue(resultSketch.getRetainedEntries() < 64);
}
- private Sketch createThetaSketch(int nominalEntries) {
- UpdateSketch updateSketch = UpdateSketch.builder().setNominalEntries(nominalEntries).build();
+ private ThetaSketch createThetaSketch(int nominalEntries) {
+ UpdatableThetaSketch updateSketch = UpdatableThetaSketch.builder().setNominalEntries(nominalEntries).build();
for (int i = 0; i < nominalEntries; i++) {
updateSketch.update(i);
}
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/aggregator/IntegerTupleSketchAggregatorTest.java b/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/aggregator/IntegerTupleSketchAggregatorTest.java
index 2dbf857fcae..241e20dd3b3 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/aggregator/IntegerTupleSketchAggregatorTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/segment/processing/aggregator/IntegerTupleSketchAggregatorTest.java
@@ -20,10 +20,10 @@ package org.apache.pinot.core.segment.processing.aggregator;
import java.util.HashMap;
import java.util.Map;
-import org.apache.datasketches.tuple.CompactSketch;
-import org.apache.datasketches.tuple.Sketch;
-import org.apache.datasketches.tuple.aninteger.IntegerSketch;
+import org.apache.datasketches.tuple.CompactTupleSketch;
+import org.apache.datasketches.tuple.TupleSketch;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
+import org.apache.datasketches.tuple.aninteger.IntegerTupleSketch;
import org.apache.pinot.core.common.ObjectSerDeUtils;
import org.apache.pinot.segment.spi.Constants;
import org.testng.annotations.BeforeMethod;
@@ -44,23 +44,23 @@ public class IntegerTupleSketchAggregatorTest {
@Test
public void testAggregateWithDefaultBehaviour() {
- Sketch<IntegerSummary> firstSketch = createTupleSketch(64);
- Sketch<IntegerSummary> secondSketch = createTupleSketch(32);
+ TupleSketch<IntegerSummary> firstSketch = createTupleSketch(64);
+ TupleSketch<IntegerSummary> secondSketch = createTupleSketch(32);
byte[] value1 = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.serialize(firstSketch);
byte[] value2 = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.serialize(secondSketch);
Map<String, String> functionParameters = new HashMap<>();
byte[] result = (byte[]) _tupleSketchAggregator.aggregate(value1, value2, functionParameters);
- Sketch<IntegerSummary> resultSketch = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(result);
+ TupleSketch<IntegerSummary> resultSketch = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(result);
assertNotNull(resultSketch);
assertEquals(resultSketch.getRetainedEntries(), 64);
}
@Test
public void testAggregateWithNominalEntries() {
- Sketch<IntegerSummary> firstSketch = createTupleSketch(64);
- Sketch<IntegerSummary> secondSketch = createTupleSketch(32);
+ TupleSketch<IntegerSummary> firstSketch = createTupleSketch(64);
+ TupleSketch<IntegerSummary> secondSketch = createTupleSketch(32);
byte[] value1 = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.serialize(firstSketch);
byte[] value2 = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.serialize(secondSketch);
@@ -69,14 +69,14 @@ public class IntegerTupleSketchAggregatorTest {
byte[] result = (byte[]) _tupleSketchAggregator.aggregate(value1, value2, functionParameters);
- Sketch<IntegerSummary> resultSketch = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(result);
+ TupleSketch<IntegerSummary> resultSketch = ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(result);
assertNotNull(resultSketch);
assertEquals(resultSketch.getRetainedEntries(), 32);
}
- private CompactSketch<IntegerSummary> createTupleSketch(int nominalEntries) {
+ private CompactTupleSketch<IntegerSummary> createTupleSketch(int nominalEntries) {
int lgK = (int) (Math.log(nominalEntries) / Math.log(2));
- IntegerSketch integerSketch = new IntegerSketch(lgK, IntegerSummary.Mode.Max);
+ IntegerTupleSketch integerSketch = new IntegerTupleSketch(lgK, IntegerSummary.Mode.Max);
for (int i = 0; i < nominalEntries; i++) {
integerSketch.update(i, 1);
}
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/startree/v2/DistinctCountIntegerSumTupleSketchStarTreeV2Test.java b/pinot-core/src/test/java/org/apache/pinot/core/startree/v2/DistinctCountIntegerSumTupleSketchStarTreeV2Test.java
index 303d748a5f3..93e1ccbfaa0 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/startree/v2/DistinctCountIntegerSumTupleSketchStarTreeV2Test.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/startree/v2/DistinctCountIntegerSumTupleSketchStarTreeV2Test.java
@@ -20,10 +20,10 @@ package org.apache.pinot.core.startree.v2;
import java.util.List;
import java.util.Random;
-import org.apache.datasketches.tuple.Sketch;
-import org.apache.datasketches.tuple.Union;
-import org.apache.datasketches.tuple.aninteger.IntegerSketch;
+import org.apache.datasketches.tuple.TupleSketch;
+import org.apache.datasketches.tuple.TupleUnion;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
+import org.apache.datasketches.tuple.aninteger.IntegerTupleSketch;
import org.apache.pinot.core.common.ObjectSerDeUtils;
import org.apache.pinot.segment.local.aggregator.IntegerTupleSketchValueAggregator;
import org.apache.pinot.segment.local.aggregator.ValueAggregator;
@@ -46,7 +46,7 @@ public class DistinctCountIntegerSumTupleSketchStarTreeV2Test extends BaseStarTr
@Override
byte[] getRandomRawValue(Random random) {
- IntegerSketch is = new IntegerSketch(4, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch is = new IntegerTupleSketch(4, IntegerSummary.Mode.Sum);
is.update(random.nextInt(100), random.nextInt(100));
return ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.serialize(is.compact());
}
@@ -60,11 +60,11 @@ public class DistinctCountIntegerSumTupleSketchStarTreeV2Test extends BaseStarTr
}
@SuppressWarnings("unchecked")
- private Sketch<IntegerSummary> toSketch(Object value) {
- if (value instanceof Union) {
- return ((Union) value).getResult();
- } else if (value instanceof Sketch) {
- return ((Sketch) value);
+ private TupleSketch<IntegerSummary> toSketch(Object value) {
+ if (value instanceof TupleUnion) {
+ return ((TupleUnion) value).getResult();
+ } else if (value instanceof TupleSketch) {
+ return ((TupleSketch) value);
} else {
throw new IllegalStateException(
"Unsupported data type for Integer Tuple Sketch aggregation: " + value.getClass().getSimpleName());
diff --git a/pinot-core/src/test/java/org/apache/pinot/core/startree/v2/DistinctCountThetaSketchStarTreeV2Test.java b/pinot-core/src/test/java/org/apache/pinot/core/startree/v2/DistinctCountThetaSketchStarTreeV2Test.java
index 07b09f064ef..85d74ccadc4 100644
--- a/pinot-core/src/test/java/org/apache/pinot/core/startree/v2/DistinctCountThetaSketchStarTreeV2Test.java
+++ b/pinot-core/src/test/java/org/apache/pinot/core/startree/v2/DistinctCountThetaSketchStarTreeV2Test.java
@@ -20,8 +20,8 @@ package org.apache.pinot.core.startree.v2;
import java.util.List;
import java.util.Random;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.Union;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.ThetaUnion;
import org.apache.pinot.segment.local.aggregator.DistinctCountThetaSketchValueAggregator;
import org.apache.pinot.segment.local.aggregator.ValueAggregator;
import org.apache.pinot.spi.data.FieldSpec.DataType;
@@ -54,11 +54,11 @@ public class DistinctCountThetaSketchStarTreeV2Test extends BaseStarTreeV2Test<O
assertEquals(toSketch(starTreeResult).getEstimate(), toSketch(nonStarTreeResult).getEstimate(), delta);
}
- private Sketch toSketch(Object value) {
- if (value instanceof Union) {
- return ((Union) value).getResult();
- } else if (value instanceof Sketch) {
- return (Sketch) value;
+ private ThetaSketch toSketch(Object value) {
+ if (value instanceof ThetaUnion) {
+ return ((ThetaUnion) value).getResult();
+ } else if (value instanceof ThetaSketch) {
+ return (ThetaSketch) value;
} else {
throw new IllegalStateException(
"Unsupported data type for Theta Sketch aggregation: " + value.getClass().getSimpleName());
diff --git a/pinot-core/src/test/java/org/apache/pinot/queries/DistinctCountThetaSketchQueriesTest.java b/pinot-core/src/test/java/org/apache/pinot/queries/DistinctCountThetaSketchQueriesTest.java
index 98e9b90900e..c5d54563423 100644
--- a/pinot-core/src/test/java/org/apache/pinot/queries/DistinctCountThetaSketchQueriesTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/queries/DistinctCountThetaSketchQueriesTest.java
@@ -26,9 +26,9 @@ import java.util.Base64;
import java.util.Iterator;
import java.util.List;
import org.apache.commons.io.FileUtils;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.UpdateSketch;
-import org.apache.datasketches.theta.UpdateSketchBuilder;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketchBuilder;
import org.apache.pinot.common.response.broker.BrokerResponseNative;
import org.apache.pinot.core.common.ObjectSerDeUtils;
import org.apache.pinot.core.operator.blocks.results.AggregationResultsBlock;
@@ -116,7 +116,7 @@ public class DistinctCountThetaSketchQueriesTest extends BaseQueriesTest {
public void setUp()
throws Exception {
FileUtils.deleteDirectory(INDEX_DIR);
- UpdateSketchBuilder sketchBuilder = new UpdateSketchBuilder();
+ UpdatableThetaSketchBuilder sketchBuilder = new UpdatableThetaSketchBuilder();
List<GenericRow> records = new ArrayList<>(NUM_RECORDS);
for (int i = 0; i < NUM_RECORDS; i++) {
@@ -133,7 +133,7 @@ public class DistinctCountThetaSketchQueriesTest extends BaseQueriesTest {
record.putValue(DOUBLE_MV_COLUMN, mvEntry);
record.putValue(STRING_MV_COLUMN, mvEntry);
// Store serialized sketches in the BYTES column
- UpdateSketch sketch = sketchBuilder.build();
+ UpdatableThetaSketch sketch = sketchBuilder.build();
sketch.update(i);
sketch.update(i + NUM_RECORDS);
sketch.update(i + 2 * NUM_RECORDS);
@@ -224,7 +224,7 @@ public class DistinctCountThetaSketchQueriesTest extends BaseQueriesTest {
List<ThetaSketchAccumulator> accumulators =
(List<ThetaSketchAccumulator>) aggregationGroupByResult.getResultForGroupId(i, groupKey._groupId);
assertEquals(accumulators.size(), 1);
- Sketch sketch = accumulators.get(0).getResult();
+ ThetaSketch sketch = accumulators.get(0).getResult();
if (i < 5) {
assertEquals(Math.round(sketch.getEstimate()), 1);
} else {
@@ -301,7 +301,8 @@ public class DistinctCountThetaSketchQueriesTest extends BaseQueriesTest {
String query = "SELECT DISTINCT_COUNT_RAW_THETA_SKETCH(intSVColumn) FROM testTable";
BrokerResponseNative brokerResponse = getBrokerResponse(query);
String serializedSketch = (String) brokerResponse.getResultTable().getRows().get(0)[0];
- Sketch sketch = ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(Base64.getDecoder().decode(serializedSketch));
+ ThetaSketch sketch =
+ ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(Base64.getDecoder().decode(serializedSketch));
assertEquals(Math.round(sketch.getEstimate()), NUM_RECORDS);
}
diff --git a/pinot-core/src/test/java/org/apache/pinot/queries/FrequentItemsSketchQueriesTest.java b/pinot-core/src/test/java/org/apache/pinot/queries/FrequentItemsSketchQueriesTest.java
index deee8f64a3f..f5b9fcae7d7 100644
--- a/pinot-core/src/test/java/org/apache/pinot/queries/FrequentItemsSketchQueriesTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/queries/FrequentItemsSketchQueriesTest.java
@@ -19,6 +19,7 @@
package org.apache.pinot.queries;
import java.io.File;
+import java.lang.foreign.MemorySegment;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
@@ -28,9 +29,8 @@ import java.util.Map;
import org.apache.commons.io.FileUtils;
import org.apache.datasketches.common.ArrayOfStringsSerDe;
import org.apache.datasketches.frequencies.ErrorType;
-import org.apache.datasketches.frequencies.ItemsSketch;
-import org.apache.datasketches.frequencies.LongsSketch;
-import org.apache.datasketches.memory.Memory;
+import org.apache.datasketches.frequencies.FrequentItemsSketch;
+import org.apache.datasketches.frequencies.FrequentLongsSketch;
import org.apache.pinot.common.response.BrokerResponse;
import org.apache.pinot.common.response.broker.ResultTable;
import org.apache.pinot.core.operator.blocks.results.AggregationResultsBlock;
@@ -123,11 +123,11 @@ public class FrequentItemsSketchQueriesTest extends BaseQueriesTest {
row.putValue(LONG_COLUMN, longValues[i]);
row.putValue(STRING_COLUMN, strValues[i]);
- LongsSketch longSketch = new LongsSketch(MAX_MAP_SIZE);
+ FrequentLongsSketch longSketch = new FrequentLongsSketch(MAX_MAP_SIZE);
longSketch.update(longValues[i]);
row.putValue(LONG_SKETCH_COLUMN, longSketch.toByteArray());
- ItemsSketch<String> strSketch = new ItemsSketch<>(MAX_MAP_SIZE);
+ FrequentItemsSketch<String> strSketch = new FrequentItemsSketch<>(MAX_MAP_SIZE);
strSketch.update(strValues[i]);
row.putValue(STRING_SKETCH_COLUMN, strSketch.toByteArray(new ArrayOfStringsSerDe()));
@@ -170,7 +170,7 @@ public class FrequentItemsSketchQueriesTest extends BaseQueriesTest {
// Fetch the exact list by count/group-by and compare
String[] exactOrdered = getExactOrderedStrings();
- assertStringsSketch((ItemsSketch<String>) aggregationResult.get(0), exactOrdered);
+ assertStringsSketch((FrequentItemsSketch<String>) aggregationResult.get(0), exactOrdered);
}
@Test
@@ -187,7 +187,7 @@ public class FrequentItemsSketchQueriesTest extends BaseQueriesTest {
// Fetch the exact list by count/group-by and compare
Long[] exactOrdered = getExactOrderedLongs();
- assertLongsSketch((LongsSketch) aggregationResult.get(0), exactOrdered);
+ assertLongsSketch((FrequentLongsSketch) aggregationResult.get(0), exactOrdered);
}
@Test
@@ -204,8 +204,8 @@ public class FrequentItemsSketchQueriesTest extends BaseQueriesTest {
assertEquals(aggregationResult.size(), 2);
// Assert the sketches are equivalent
- ItemsSketch<String> sketch1 = (ItemsSketch<String>) aggregationResult.get(0);
- ItemsSketch<String> sketch2 = (ItemsSketch<String>) aggregationResult.get(1);
+ FrequentItemsSketch<String> sketch1 = (FrequentItemsSketch<String>) aggregationResult.get(0);
+ FrequentItemsSketch<String> sketch2 = (FrequentItemsSketch<String>) aggregationResult.get(1);
assertEquals(
sketch1.getFrequentItems(ErrorType.NO_FALSE_NEGATIVES),
sketch2.getFrequentItems(ErrorType.NO_FALSE_NEGATIVES));
@@ -225,8 +225,8 @@ public class FrequentItemsSketchQueriesTest extends BaseQueriesTest {
assertEquals(aggregationResult.size(), 2);
// Assert the sketches are equivalent
- LongsSketch sketch1 = (LongsSketch) aggregationResult.get(0);
- LongsSketch sketch2 = (LongsSketch) aggregationResult.get(1);
+ FrequentLongsSketch sketch1 = (FrequentLongsSketch) aggregationResult.get(0);
+ FrequentLongsSketch sketch2 = (FrequentLongsSketch) aggregationResult.get(1);
assertEquals(
sketch1.getFrequentItems(ErrorType.NO_FALSE_NEGATIVES),
sketch2.getFrequentItems(ErrorType.NO_FALSE_NEGATIVES));
@@ -249,7 +249,7 @@ public class FrequentItemsSketchQueriesTest extends BaseQueriesTest {
Map<String, ArrayList<String>> exactOrdered = getExactOrderedStringGroups();
for (Object[] row: rows) {
String group = (String) row[0];
- ItemsSketch<String> sketch = decodeStringsSketch((String) row[1]);
+ FrequentItemsSketch<String> sketch = decodeStringsSketch((String) row[1]);
List<String> exactOrder = exactOrdered.get(group);
assertStringsSketch(sketch, exactOrder);
}
@@ -272,7 +272,7 @@ public class FrequentItemsSketchQueriesTest extends BaseQueriesTest {
Map<String, ArrayList<Long>> exactOrdered = getExactOrderedLongGroups();
for (Object[] row: rows) {
String group = (String) row[0];
- LongsSketch sketch = decodeLongsSketch((String) row[1]);
+ FrequentLongsSketch sketch = decodeLongsSketch((String) row[1]);
List<Long> exactOrder = exactOrdered.get(group);
assertLongsSketch(sketch, exactOrder);
}
@@ -341,42 +341,42 @@ public class FrequentItemsSketchQueriesTest extends BaseQueriesTest {
return order;
}
- private void assertStringsSketch(ItemsSketch<String> sketch, List<String> exact) {
+ private void assertStringsSketch(FrequentItemsSketch<String> sketch, List<String> exact) {
String[] arr = new String[exact.size()];
exact.toArray(arr);
assertStringsSketch(sketch, arr);
}
- private void assertStringsSketch(ItemsSketch<String> sketch, String[] exact) {
- ItemsSketch.Row[] items = sketch.getFrequentItems(ErrorType.NO_FALSE_NEGATIVES);
+ private void assertStringsSketch(FrequentItemsSketch<String> sketch, String[] exact) {
+ FrequentItemsSketch.Row[] items = sketch.getFrequentItems(ErrorType.NO_FALSE_NEGATIVES);
assertEquals(exact.length, items.length);
for (int i = 0; i < exact.length; i++) {
assertEquals((String) items[i].getItem(), exact[i]);
}
}
- private void assertLongsSketch(LongsSketch sketch, List<Long> exact) {
+ private void assertLongsSketch(FrequentLongsSketch sketch, List<Long> exact) {
Long[] arr = new Long[exact.size()];
exact.toArray(arr);
assertLongsSketch(sketch, arr);
}
- private void assertLongsSketch(LongsSketch sketch, Long[] exact) {
- LongsSketch.Row[] items = sketch.getFrequentItems(ErrorType.NO_FALSE_NEGATIVES);
+ private void assertLongsSketch(FrequentLongsSketch sketch, Long[] exact) {
+ FrequentLongsSketch.Row[] items = sketch.getFrequentItems(ErrorType.NO_FALSE_NEGATIVES);
assertEquals(exact.length, items.length);
for (int i = 0; i < exact.length; i++) {
assertEquals((Long) items[i].getItem(), exact[i]);
}
}
- private ItemsSketch<String> decodeStringsSketch(String encodedSketch) {
+ private FrequentItemsSketch<String> decodeStringsSketch(String encodedSketch) {
byte[] byteArr = Base64.getDecoder().decode(encodedSketch);
- return ItemsSketch.getInstance(Memory.wrap(byteArr), new ArrayOfStringsSerDe());
+ return FrequentItemsSketch.getInstance(MemorySegment.ofArray(byteArr), new ArrayOfStringsSerDe());
}
- private LongsSketch decodeLongsSketch(String encodedSketch) {
+ private FrequentLongsSketch decodeLongsSketch(String encodedSketch) {
byte[] byteArr = Base64.getDecoder().decode(encodedSketch);
- return LongsSketch.getInstance(Memory.wrap(byteArr));
+ return FrequentLongsSketch.getInstance(MemorySegment.ofArray(byteArr));
}
@AfterClass
diff --git a/pinot-core/src/test/java/org/apache/pinot/queries/FunnelCountQueriesThetaSketchTest.java b/pinot-core/src/test/java/org/apache/pinot/queries/FunnelCountQueriesThetaSketchTest.java
index fe9b6380ff4..ca7edee75d0 100644
--- a/pinot-core/src/test/java/org/apache/pinot/queries/FunnelCountQueriesThetaSketchTest.java
+++ b/pinot-core/src/test/java/org/apache/pinot/queries/FunnelCountQueriesThetaSketchTest.java
@@ -19,7 +19,7 @@
package org.apache.pinot.queries;
import java.util.List;
-import org.apache.datasketches.theta.Sketch;
+import org.apache.datasketches.theta.ThetaSketch;
import org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImplTestUtils;
import org.apache.pinot.segment.spi.IndexSegment;
import org.apache.pinot.segment.spi.MutableSegment;
@@ -63,7 +63,7 @@ public class FunnelCountQueriesThetaSketchTest extends BaseFunnelCountQueriesTes
@Override
protected void assertIntermediateResult(Object intermediateResult, long[] expectedCounts) {
assertTrue(intermediateResult instanceof List);
- List<Sketch> sketches = (List<Sketch>) intermediateResult;
+ List<ThetaSketch> sketches = (List<ThetaSketch>) intermediateResult;
// First step should match
assertEquals(Math.round(sketches.get(0).getEstimate()), expectedCounts[0]);
for (int i = 1; i < sketches.size(); i++) {
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/ThetaSketchTest.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/ThetaSketchTest.java
index 04f2964de89..49aeb00c076 100644
--- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/ThetaSketchTest.java
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/ThetaSketchTest.java
@@ -30,8 +30,8 @@ import org.apache.avro.file.DataFileWriter;
import org.apache.avro.generic.GenericData;
import org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.commons.lang3.tuple.Pair;
-import org.apache.datasketches.theta.UpdateSketch;
-import org.apache.datasketches.theta.UpdateSketchBuilder;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketchBuilder;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
import org.testng.annotations.Test;
@@ -124,7 +124,7 @@ public class ThetaSketchTest extends CustomDataQueryClusterIntegrationTest {
for (String gender : allGenders) {
// calculate theta sketch
- UpdateSketch sketch = new UpdateSketchBuilder().build();
+ UpdatableThetaSketch sketch = new UpdatableThetaSketchBuilder().build();
genderCourseToStudentIds.forEach((genderCourse, studentIds) -> {
if (gender.equals(genderCourse.getLeft())) {
studentIds.forEach(sketch::update);
@@ -147,7 +147,7 @@ public class ThetaSketchTest extends CustomDataQueryClusterIntegrationTest {
for (String course : allCountries) {
// calculate theta sketch
- UpdateSketch sketch = new UpdateSketchBuilder().build();
+ UpdatableThetaSketch sketch = new UpdatableThetaSketchBuilder().build();
genderCourseToStudentIds.forEach((genderCourse, studentIds) -> {
if (course.equals(genderCourse.getRight())) {
studentIds.forEach(sketch::update);
diff --git a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/TupleSketchTest.java b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/TupleSketchTest.java
index e7408295f11..15789c501bf 100644
--- a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/TupleSketchTest.java
+++ b/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/custom/TupleSketchTest.java
@@ -25,11 +25,11 @@ import java.util.Base64;
import java.util.List;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.generic.GenericData;
-import org.apache.datasketches.tuple.Intersection;
-import org.apache.datasketches.tuple.Sketch;
-import org.apache.datasketches.tuple.aninteger.IntegerSketch;
+import org.apache.datasketches.tuple.TupleIntersection;
+import org.apache.datasketches.tuple.TupleSketch;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.datasketches.tuple.aninteger.IntegerSummarySetOperations;
+import org.apache.datasketches.tuple.aninteger.IntegerTupleSketch;
import org.apache.pinot.core.common.ObjectSerDeUtils;
import org.apache.pinot.spi.data.FieldSpec;
import org.apache.pinot.spi.data.Schema;
@@ -64,7 +64,7 @@ public class TupleSketchTest extends CustomDataQueryClusterIntegrationTest {
JsonNode jsonNode = postQuery(query);
long distinctCount = jsonNode.get("resultTable").get("rows").get(0).get(0).asLong();
byte[] rawSketchBytes = Base64.getDecoder().decode(jsonNode.get("resultTable").get("rows").get(0).get(1).asText());
- Sketch<IntegerSummary> deserializedSketch =
+ TupleSketch<IntegerSummary> deserializedSketch =
ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(rawSketchBytes);
assertTrue(distinctCount > 0);
@@ -149,11 +149,11 @@ public class TupleSketchTest extends CustomDataQueryClusterIntegrationTest {
String sketch1 = jsonNode.get("resultTable").get("rows").get(0).get(1).asText();
String sketch2 = jsonNode.get("resultTable").get("rows").get(0).get(2).asText();
- Sketch<IntegerSummary> deserializedSketch1 =
+ TupleSketch<IntegerSummary> deserializedSketch1 =
ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(Base64.getDecoder().decode(sketch1));
- Sketch<IntegerSummary> deserializedSketch2 =
+ TupleSketch<IntegerSummary> deserializedSketch2 =
ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(Base64.getDecoder().decode(sketch2));
- Intersection<IntegerSummary> intersection = new Intersection<>(new IntegerSummarySetOperations(
+ TupleIntersection<IntegerSummary> intersection = new TupleIntersection<>(new IntegerSummarySetOperations(
IntegerSummary.Mode.Sum, IntegerSummary.Mode.Sum));
intersection.intersect(deserializedSketch1);
intersection.intersect(deserializedSketch2);
@@ -189,7 +189,7 @@ public class TupleSketchTest extends CustomDataQueryClusterIntegrationTest {
JsonNode jsonNode = postQuery(query);
long distinctCount = jsonNode.get("resultTable").get("rows").get(0).get(0).asLong();
byte[] rawSketchBytes = Base64.getDecoder().decode(jsonNode.get("resultTable").get("rows").get(0).get(1).asText());
- Sketch<IntegerSummary> deserializedSketch =
+ TupleSketch<IntegerSummary> deserializedSketch =
ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(rawSketchBytes);
assertTrue(distinctCount > 0);
@@ -224,7 +224,7 @@ public class TupleSketchTest extends CustomDataQueryClusterIntegrationTest {
JsonNode jsonNode = postQuery(query);
long distinctCount = jsonNode.get("resultTable").get("rows").get(0).get(0).asLong();
byte[] rawSketchBytes = Base64.getDecoder().decode(jsonNode.get("resultTable").get("rows").get(0).get(1).asText());
- Sketch<IntegerSummary> deserializedSketch =
+ TupleSketch<IntegerSummary> deserializedSketch =
ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(rawSketchBytes);
assertTrue(distinctCount > 0);
assertEquals(Double.valueOf(deserializedSketch.getEstimate()).longValue(), distinctCount);
@@ -302,7 +302,7 @@ public class TupleSketchTest extends CustomDataQueryClusterIntegrationTest {
}
private byte[] getRandomRawValue() {
- IntegerSketch is = new IntegerSketch(4, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch is = new IntegerTupleSketch(4, IntegerSummary.Mode.Sum);
is.update(RANDOM.nextInt(100), RANDOM.nextInt(100));
return ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.serialize(is.compact());
}
diff --git a/pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkSketchSerDe.java b/pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkSketchSerDe.java
new file mode 100644
index 00000000000..63d6f3b9d6b
--- /dev/null
+++ b/pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkSketchSerDe.java
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.perf;
+
+import java.util.concurrent.TimeUnit;
+import org.apache.datasketches.cpc.CpcSketch;
+import org.apache.datasketches.cpc.CpcUnion;
+import org.apache.datasketches.kll.KllDoublesSketch;
+import org.apache.datasketches.theta.CompactThetaSketch;
+import org.apache.datasketches.theta.ThetaSetOperationBuilder;
+import org.apache.datasketches.theta.ThetaUnion;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
+import org.apache.datasketches.tuple.TupleUnion;
+import org.apache.datasketches.tuple.aninteger.IntegerSummary;
+import org.apache.datasketches.tuple.aninteger.IntegerSummarySetOperations;
+import org.apache.datasketches.tuple.aninteger.IntegerTupleSketch;
+import org.apache.pinot.core.common.ObjectSerDeUtils;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+
+/// Benchmarks the sketch [ObjectSerDeUtils.ObjectSerDe] deserialize + merge hot paths that run once per row on
+/// serialized-sketch BYTES columns during aggregation. Used to compare the datasketches-java memory access layer
+/// across library upgrades (e.g. 6.2.0 Unsafe-based reads vs 9.0.0 java.lang.foreign MemorySegment reads).
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+@Fork(1)
+@Warmup(iterations = 3, time = 2)
+@Measurement(iterations = 5, time = 2)
+@State(Scope.Benchmark)
+public class BenchmarkSketchSerDe {
+ private static final int NUM_SKETCHES = 64;
+ private static final int NOMINAL_ENTRIES = 4096;
+ private static final int LG_K = 12;
+
+ @Param({"1000", "50000"})
+ private int _valuesPerSketch;
+
+ private byte[][] _thetaBytes;
+ private byte[][] _tupleBytes;
+ private byte[][] _kllBytes;
+ private byte[][] _cpcBytes;
+ private CompactThetaSketch _thetaSketch;
+
+ @Setup
+ public void setUp() {
+ _thetaBytes = new byte[NUM_SKETCHES][];
+ _tupleBytes = new byte[NUM_SKETCHES][];
+ _kllBytes = new byte[NUM_SKETCHES][];
+ _cpcBytes = new byte[NUM_SKETCHES][];
+ for (int i = 0; i < NUM_SKETCHES; i++) {
+ UpdatableThetaSketch theta = UpdatableThetaSketch.builder().setNominalEntries(NOMINAL_ENTRIES).build();
+ IntegerTupleSketch tuple = new IntegerTupleSketch(LG_K, IntegerSummary.Mode.Sum);
+ KllDoublesSketch kll = KllDoublesSketch.newHeapInstance();
+ CpcSketch cpc = new CpcSketch(LG_K);
+ // Overlapping key ranges so unions do real merge work
+ long base = (long) i * _valuesPerSketch / 2;
+ for (int j = 0; j < _valuesPerSketch; j++) {
+ long key = base + j;
+ theta.update(key);
+ tuple.update(key, 1);
+ kll.update(key);
+ cpc.update(key);
+ }
+ _thetaBytes[i] = theta.compact().toByteArray();
+ _tupleBytes[i] = tuple.compact().toByteArray();
+ _kllBytes[i] = kll.toByteArray();
+ _cpcBytes[i] = cpc.toByteArray();
+ }
+ UpdatableThetaSketch serializeSource = UpdatableThetaSketch.builder().setNominalEntries(NOMINAL_ENTRIES).build();
+ for (int j = 0; j < _valuesPerSketch; j++) {
+ serializeSource.update(j);
+ }
+ _thetaSketch = serializeSource.compact();
+ }
+
+ @Benchmark
+ public double thetaDeserializeAndUnion() {
+ ThetaUnion union = new ThetaSetOperationBuilder().setNominalEntries(NOMINAL_ENTRIES).buildUnion();
+ for (byte[] bytes : _thetaBytes) {
+ union.union(ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(bytes));
+ }
+ return union.getResult().getEstimate();
+ }
+
+ @Benchmark
+ public double tupleDeserializeAndUnion() {
+ TupleUnion<IntegerSummary> union =
+ new TupleUnion<>(NOMINAL_ENTRIES, new IntegerSummarySetOperations(IntegerSummary.Mode.Sum,
+ IntegerSummary.Mode.Sum));
+ for (byte[] bytes : _tupleBytes) {
+ union.union(ObjectSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(bytes));
+ }
+ return union.getResult().getEstimate();
+ }
+
+ @Benchmark
+ public double kllDeserializeAndMerge() {
+ KllDoublesSketch merged = KllDoublesSketch.newHeapInstance();
+ for (byte[] bytes : _kllBytes) {
+ merged.merge(ObjectSerDeUtils.KLL_SKETCH_SER_DE.deserialize(bytes));
+ }
+ return merged.getQuantile(0.5);
+ }
+
+ @Benchmark
+ public double cpcDeserializeAndUnion() {
+ CpcUnion union = new CpcUnion(LG_K);
+ for (byte[] bytes : _cpcBytes) {
+ union.update(ObjectSerDeUtils.DATA_SKETCH_CPC_SER_DE.deserialize(bytes));
+ }
+ return union.getResult().getEstimate();
+ }
+
+ @Benchmark
+ public int thetaSerialize() {
+ return ObjectSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(_thetaSketch).length;
+ }
+
+ public static void main(String[] args)
+ throws Exception {
+ new Runner(new OptionsBuilder().include(BenchmarkSketchSerDe.class.getSimpleName()).build()).run();
+ }
+}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/aggregator/DistinctCountThetaSketchValueAggregator.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/aggregator/DistinctCountThetaSketchValueAggregator.java
index 5dba35d637c..c735ea4f906 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/aggregator/DistinctCountThetaSketchValueAggregator.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/aggregator/DistinctCountThetaSketchValueAggregator.java
@@ -20,9 +20,9 @@ package org.apache.pinot.segment.local.aggregator;
import com.google.common.annotations.VisibleForTesting;
import java.util.List;
-import org.apache.datasketches.theta.SetOperationBuilder;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.Union;
+import org.apache.datasketches.theta.ThetaSetOperationBuilder;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.ThetaUnion;
import org.apache.pinot.common.request.context.ExpressionContext;
import org.apache.pinot.segment.local.utils.CustomSerDeUtils;
import org.apache.pinot.segment.spi.AggregationFunctionType;
@@ -33,7 +33,7 @@ import org.apache.pinot.spi.utils.CommonConstants;
public class DistinctCountThetaSketchValueAggregator implements ValueAggregator<Object, Object> {
public static final DataType AGGREGATED_VALUE_TYPE = DataType.BYTES;
- private final SetOperationBuilder _setOperationBuilder;
+ private final ThetaSetOperationBuilder _setOperationBuilder;
private final int _nominalEntries;
// This changes a lot similar to the Bitmap aggregator
@@ -46,7 +46,7 @@ public class DistinctCountThetaSketchValueAggregator implements ValueAggregator<
} else {
_nominalEntries = arguments.get(0).getLiteral().getIntValue();
}
- _setOperationBuilder = Union.builder().setNominalEntries(_nominalEntries);
+ _setOperationBuilder = ThetaUnion.builder().setNominalEntries(_nominalEntries);
}
@Override
@@ -64,7 +64,7 @@ public class DistinctCountThetaSketchValueAggregator implements ValueAggregator<
return _nominalEntries;
}
- private void singleItemUpdate(Union thetaUnion, Object rawValue) {
+ private void singleItemUpdate(ThetaUnion thetaUnion, Object rawValue) {
if (rawValue instanceof String) {
thetaUnion.update((String) rawValue);
} else if (rawValue instanceof Integer) {
@@ -77,17 +77,17 @@ public class DistinctCountThetaSketchValueAggregator implements ValueAggregator<
thetaUnion.update((Float) rawValue);
} else if (rawValue instanceof Object[]) {
multiItemUpdate(thetaUnion, (Object[]) rawValue);
- } else if (rawValue instanceof Sketch) {
- thetaUnion.union((Sketch) rawValue);
- } else if (rawValue instanceof Union) {
- thetaUnion.union(((Union) rawValue).getResult());
+ } else if (rawValue instanceof ThetaSketch) {
+ thetaUnion.union((ThetaSketch) rawValue);
+ } else if (rawValue instanceof ThetaUnion) {
+ thetaUnion.union(((ThetaUnion) rawValue).getResult());
} else {
throw new IllegalStateException(
"Unsupported data type for Theta Sketch aggregation: " + rawValue.getClass().getSimpleName());
}
}
- private void multiItemUpdate(Union thetaUnion, Object[] rawValues) {
+ private void multiItemUpdate(ThetaUnion thetaUnion, Object[] rawValues) {
if (rawValues instanceof String[]) {
for (String s : (String[]) rawValues) {
thetaUnion.update(s);
@@ -118,10 +118,10 @@ public class DistinctCountThetaSketchValueAggregator implements ValueAggregator<
public Object getInitialAggregatedValue(Object rawValue) {
// NOTE: rawValue cannot be null because this aggregator can only be used for star-tree index.
assert rawValue != null;
- Union thetaUnion = _setOperationBuilder.buildUnion();
- if (rawValue instanceof byte[]) { // Serialized Sketch
+ ThetaUnion thetaUnion = _setOperationBuilder.buildUnion();
+ if (rawValue instanceof byte[]) { // Serialized ThetaSketch
byte[] bytes = (byte[]) rawValue;
- Sketch sketch = deserializeAggregatedValue(bytes);
+ ThetaSketch sketch = deserializeAggregatedValue(bytes);
thetaUnion.union(sketch);
} else if (rawValue instanceof byte[][]) { // Multiple Serialized Sketches
byte[][] serializedSketches = (byte[][]) rawValue;
@@ -135,14 +135,14 @@ public class DistinctCountThetaSketchValueAggregator implements ValueAggregator<
return thetaUnion;
}
- private Union extractUnion(Object value) {
+ private ThetaUnion extractUnion(Object value) {
if (value == null) {
return _setOperationBuilder.buildUnion();
- } else if (value instanceof Union) {
- return (Union) value;
- } else if (value instanceof Sketch) {
- Sketch sketch = (Sketch) value;
- Union thetaUnion = _setOperationBuilder.buildUnion();
+ } else if (value instanceof ThetaUnion) {
+ return (ThetaUnion) value;
+ } else if (value instanceof ThetaSketch) {
+ ThetaSketch sketch = (ThetaSketch) value;
+ ThetaUnion thetaUnion = _setOperationBuilder.buildUnion();
thetaUnion.union(sketch);
return thetaUnion;
} else {
@@ -153,9 +153,9 @@ public class DistinctCountThetaSketchValueAggregator implements ValueAggregator<
@Override
public Object applyRawValue(Object aggregatedValue, Object rawValue) {
- Union thetaUnion = extractUnion(aggregatedValue);
+ ThetaUnion thetaUnion = extractUnion(aggregatedValue);
if (rawValue instanceof byte[]) {
- Sketch sketch = deserializeAggregatedValue((byte[]) rawValue);
+ ThetaSketch sketch = deserializeAggregatedValue((byte[]) rawValue);
thetaUnion.union(sketch);
} else {
singleItemUpdate(thetaUnion, rawValue);
@@ -166,7 +166,7 @@ public class DistinctCountThetaSketchValueAggregator implements ValueAggregator<
@Override
public Object applyAggregatedValue(Object value, Object aggregatedValue) {
- Union thetaUnion = extractUnion(aggregatedValue);
+ ThetaUnion thetaUnion = extractUnion(aggregatedValue);
singleItemUpdate(thetaUnion, value);
_maxByteSize = Math.max(_maxByteSize, thetaUnion.getCurrentBytes());
return thetaUnion;
@@ -189,10 +189,10 @@ public class DistinctCountThetaSketchValueAggregator implements ValueAggregator<
@Override
public byte[] serializeAggregatedValue(Object value) {
- if (value instanceof Union) {
- return CustomSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(((Union) value).getResult());
- } else if (value instanceof Sketch) {
- return CustomSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(((Sketch) value));
+ if (value instanceof ThetaUnion) {
+ return CustomSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(((ThetaUnion) value).getResult());
+ } else if (value instanceof ThetaSketch) {
+ return CustomSerDeUtils.DATA_SKETCH_THETA_SER_DE.serialize(((ThetaSketch) value));
} else {
throw new IllegalStateException(
"Unsupported data type for Theta Sketch aggregation: " + value.getClass().getSimpleName());
@@ -200,7 +200,7 @@ public class DistinctCountThetaSketchValueAggregator implements ValueAggregator<
}
@Override
- public Sketch deserializeAggregatedValue(byte[] bytes) {
+ public ThetaSketch deserializeAggregatedValue(byte[] bytes) {
return CustomSerDeUtils.DATA_SKETCH_THETA_SER_DE.deserialize(bytes);
}
}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/aggregator/IntegerTupleSketchValueAggregator.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/aggregator/IntegerTupleSketchValueAggregator.java
index e5db940dbcd..d25ec2c2f18 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/aggregator/IntegerTupleSketchValueAggregator.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/aggregator/IntegerTupleSketchValueAggregator.java
@@ -21,8 +21,8 @@ package org.apache.pinot.segment.local.aggregator;
import com.google.common.annotations.VisibleForTesting;
import java.util.List;
import javax.annotation.Nullable;
-import org.apache.datasketches.tuple.Sketch;
-import org.apache.datasketches.tuple.Union;
+import org.apache.datasketches.tuple.TupleSketch;
+import org.apache.datasketches.tuple.TupleUnion;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.datasketches.tuple.aninteger.IntegerSummarySetOperations;
import org.apache.pinot.common.request.context.ExpressionContext;
@@ -62,23 +62,25 @@ public class IntegerTupleSketchValueAggregator implements ValueAggregator<byte[]
@Override
public Object getInitialAggregatedValue(@Nullable byte[] rawValue) {
- Union tupleUnion = new Union<>(_nominalEntries, new IntegerSummarySetOperations(_mode, _mode));
+ TupleUnion<IntegerSummary> tupleUnion =
+ new TupleUnion<>(_nominalEntries, new IntegerSummarySetOperations(_mode, _mode));
if (rawValue == null) {
return tupleUnion;
}
- Sketch<IntegerSummary> initialValue = deserializeAggregatedValue(rawValue);
+ TupleSketch<IntegerSummary> initialValue = deserializeAggregatedValue(rawValue);
tupleUnion.union(initialValue);
return tupleUnion;
}
- private Union extractUnion(Object value) {
+ private TupleUnion<IntegerSummary> extractUnion(Object value) {
if (value == null) {
- return new Union<>(_nominalEntries, new IntegerSummarySetOperations(_mode, _mode));
- } else if (value instanceof Union) {
- return (Union) value;
- } else if (value instanceof Sketch) {
- Sketch sketch = (Sketch) value;
- Union tupleUnion = new Union<>(_nominalEntries, new IntegerSummarySetOperations(_mode, _mode));
+ return new TupleUnion<>(_nominalEntries, new IntegerSummarySetOperations(_mode, _mode));
+ } else if (value instanceof TupleUnion) {
+ return (TupleUnion<IntegerSummary>) value;
+ } else if (value instanceof TupleSketch) {
+ TupleSketch<IntegerSummary> sketch = (TupleSketch<IntegerSummary>) value;
+ TupleUnion<IntegerSummary> tupleUnion =
+ new TupleUnion<>(_nominalEntries, new IntegerSummarySetOperations(_mode, _mode));
tupleUnion.union(sketch);
return tupleUnion;
} else {
@@ -87,11 +89,11 @@ public class IntegerTupleSketchValueAggregator implements ValueAggregator<byte[]
}
}
- private Sketch extractSketch(Object value) {
- if (value instanceof Union) {
- return ((Union) value).getResult();
- } else if (value instanceof Sketch) {
- return ((Sketch) value);
+ private TupleSketch<IntegerSummary> extractSketch(Object value) {
+ if (value instanceof TupleUnion) {
+ return ((TupleUnion<IntegerSummary>) value).getResult();
+ } else if (value instanceof TupleSketch) {
+ return ((TupleSketch<IntegerSummary>) value);
} else {
throw new IllegalStateException(
"Unsupported data type for Integer Tuple Sketch aggregation: " + value.getClass().getSimpleName());
@@ -100,15 +102,15 @@ public class IntegerTupleSketchValueAggregator implements ValueAggregator<byte[]
@Override
public Object applyRawValue(Object aggregatedValue, byte[] rawValue) {
- Union tupleUnion = extractUnion(aggregatedValue);
+ TupleUnion<IntegerSummary> tupleUnion = extractUnion(aggregatedValue);
tupleUnion.union(deserializeAggregatedValue(rawValue));
return tupleUnion;
}
@Override
public Object applyAggregatedValue(Object value, Object aggregatedValue) {
- Union tupleUnion = extractUnion(aggregatedValue);
- Sketch sketch = extractSketch(value);
+ TupleUnion<IntegerSummary> tupleUnion = extractUnion(aggregatedValue);
+ TupleSketch<IntegerSummary> sketch = extractSketch(value);
tupleUnion.union(sketch);
return tupleUnion;
}
@@ -145,12 +147,12 @@ public class IntegerTupleSketchValueAggregator implements ValueAggregator<byte[]
@Override
public byte[] serializeAggregatedValue(Object value) {
- Sketch sketch = extractSketch(value);
+ TupleSketch<IntegerSummary> sketch = extractSketch(value);
return sketch.compact().toByteArray();
}
@Override
- public Sketch<IntegerSummary> deserializeAggregatedValue(byte[] bytes) {
+ public TupleSketch<IntegerSummary> deserializeAggregatedValue(byte[] bytes) {
return CustomSerDeUtils.DATA_SKETCH_INT_TUPLE_SER_DE.deserialize(bytes);
}
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedFrequentLongsSketch.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedFrequentLongsSketch.java
index 203515c67bb..2f0397fb5c0 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedFrequentLongsSketch.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedFrequentLongsSketch.java
@@ -19,21 +19,21 @@
package org.apache.pinot.segment.local.customobject;
import java.util.Base64;
-import org.apache.datasketches.frequencies.LongsSketch;
+import org.apache.datasketches.frequencies.FrequentLongsSketch;
-public class SerializedFrequentLongsSketch implements Comparable<LongsSketch> {
- private final LongsSketch _sketch;
+public class SerializedFrequentLongsSketch implements Comparable<SerializedFrequentLongsSketch> {
+ private final FrequentLongsSketch _sketch;
- public SerializedFrequentLongsSketch(LongsSketch sketch) {
+ public SerializedFrequentLongsSketch(FrequentLongsSketch sketch) {
_sketch = sketch;
}
@Override
- public int compareTo(LongsSketch other) {
+ public int compareTo(SerializedFrequentLongsSketch other) {
// There is no well-defined ordering for these sketches
// numActiveItems is just a placeholder, which can be changed later
- return _sketch.getNumActiveItems() - other.getNumActiveItems();
+ return Integer.compare(_sketch.getNumActiveItems(), other._sketch.getNumActiveItems());
}
@Override
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedFrequentStringsSketch.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedFrequentStringsSketch.java
index 040692a553f..3f05e08b502 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedFrequentStringsSketch.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/SerializedFrequentStringsSketch.java
@@ -20,20 +20,20 @@ package org.apache.pinot.segment.local.customobject;
import java.util.Base64;
import org.apache.datasketches.common.ArrayOfStringsSerDe;
-import org.apache.datasketches.frequencies.ItemsSketch;
+import org.apache.datasketches.frequencies.FrequentItemsSketch;
-public class SerializedFrequentStringsSketch implements Comparable<ItemsSketch<String>> {
- private final ItemsSketch<String> _sketch;
+public class SerializedFrequentStringsSketch implements Comparable<SerializedFrequentStringsSketch> {
+ private final FrequentItemsSketch<String> _sketch;
- public SerializedFrequentStringsSketch(ItemsSketch<String> sketch) {
+ public SerializedFrequentStringsSketch(FrequentItemsSketch<String> sketch) {
_sketch = sketch;
}
@Override
- public int compareTo(ItemsSketch<String> other) {
+ public int compareTo(SerializedFrequentStringsSketch other) {
// There is no well-defined ordering for these sketches
// numActiveItems is just a placeholder, which can be changed later
- return _sketch.getNumActiveItems() - other.getNumActiveItems();
+ return Integer.compare(_sketch.getNumActiveItems(), other._sketch.getNumActiveItems());
}
@Override
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/ThetaSketchAccumulator.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/ThetaSketchAccumulator.java
index ee8f8ba05c2..c7a7d37d3cd 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/ThetaSketchAccumulator.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/ThetaSketchAccumulator.java
@@ -19,9 +19,9 @@
package org.apache.pinot.segment.local.customobject;
import java.util.Comparator;
-import org.apache.datasketches.theta.SetOperationBuilder;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.Union;
+import org.apache.datasketches.theta.ThetaSetOperationBuilder;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.ThetaUnion;
/**
@@ -33,9 +33,9 @@ import org.apache.datasketches.theta.Union;
* that fall below the minimum Theta value for all input sketches ("Broder Rule"). When the initial Theta value is
* set to the minimum immediately, further gains can be realised.
*/
-public class ThetaSketchAccumulator extends CustomObjectAccumulator<Sketch> {
- private SetOperationBuilder _setOperationBuilder = new SetOperationBuilder();
- private Union _union;
+public class ThetaSketchAccumulator extends CustomObjectAccumulator<ThetaSketch> {
+ private ThetaSetOperationBuilder _setOperationBuilder = new ThetaSetOperationBuilder();
+ private ThetaUnion _union;
public ThetaSketchAccumulator() {
}
@@ -44,21 +44,21 @@ public class ThetaSketchAccumulator extends CustomObjectAccumulator<Sketch> {
// happens on serialization. Therefore, when deserialized, the values may be null and will
// require re-initialisation. Since the primary use case is at query time for the Broker
// and Server, these properties are already in memory and are re-set.
- public ThetaSketchAccumulator(SetOperationBuilder setOperationBuilder, int threshold) {
+ public ThetaSketchAccumulator(ThetaSetOperationBuilder setOperationBuilder, int threshold) {
super(threshold);
_setOperationBuilder = setOperationBuilder;
}
- public void setSetOperationBuilder(SetOperationBuilder setOperationBuilder) {
+ public void setSetOperationBuilder(ThetaSetOperationBuilder setOperationBuilder) {
_setOperationBuilder = setOperationBuilder;
}
@Override
- public Sketch getResult() {
+ public ThetaSketch getResult() {
return unionAll();
}
- private Sketch unionAll() {
+ private ThetaSketch unionAll() {
if (_union == null) {
_union = _setOperationBuilder.buildUnion();
}
@@ -75,7 +75,7 @@ public class ThetaSketchAccumulator extends CustomObjectAccumulator<Sketch> {
}
// Performance optimization: ensure that the minimum Theta is used for "early stop".
- // The "early stop" optimization is implemented in the Apache Datasketches Union operation for
+ // The "early stop" optimization is implemented in the Apache Datasketches ThetaUnion operation for
// ordered and compact Theta sketches. Internally, a compact and ordered Theta sketch can be
// compared to a sorted array of K items. When performing a union, only those items from
// the input sketch less than Theta need to be processed. The loop terminates as soon as a hash
@@ -83,8 +83,8 @@ public class ThetaSketchAccumulator extends CustomObjectAccumulator<Sketch> {
// The following "sort" improves on this further by selecting the minimal Theta value up-front,
// which results in fewer redundant entries being retained and subsequently discarded during the
// union operation.
- _accumulator.sort(Comparator.comparingDouble(Sketch::getTheta));
- for (Sketch accumulatedSketch : _accumulator) {
+ _accumulator.sort(Comparator.comparingDouble(ThetaSketch::getTheta));
+ for (ThetaSketch accumulatedSketch : _accumulator) {
_union.union(accumulatedSketch);
}
_accumulator.clear();
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/TupleIntSketchAccumulator.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/TupleIntSketchAccumulator.java
index de3c419be36..b301b160b00 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/TupleIntSketchAccumulator.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/customobject/TupleIntSketchAccumulator.java
@@ -19,8 +19,8 @@
package org.apache.pinot.segment.local.customobject;
import java.util.Comparator;
-import org.apache.datasketches.tuple.Sketch;
-import org.apache.datasketches.tuple.Union;
+import org.apache.datasketches.tuple.TupleSketch;
+import org.apache.datasketches.tuple.TupleUnion;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.datasketches.tuple.aninteger.IntegerSummarySetOperations;
@@ -35,10 +35,10 @@ import org.apache.datasketches.tuple.aninteger.IntegerSummarySetOperations;
* that fall below the minimum Theta value for all input sketches ("Broder Rule"). When the initial Theta value is
* set to the minimum immediately, further gains can be realised.
*/
-public class TupleIntSketchAccumulator extends CustomObjectAccumulator<Sketch<IntegerSummary>> {
+public class TupleIntSketchAccumulator extends CustomObjectAccumulator<TupleSketch<IntegerSummary>> {
private IntegerSummarySetOperations _setOperations;
private int _nominalEntries;
- private Union<IntegerSummary> _union;
+ private TupleUnion<IntegerSummary> _union;
public TupleIntSketchAccumulator() {
}
@@ -62,13 +62,13 @@ public class TupleIntSketchAccumulator extends CustomObjectAccumulator<Sketch<In
}
@Override
- public Sketch<IntegerSummary> getResult() {
+ public TupleSketch<IntegerSummary> getResult() {
return unionAll();
}
- private Sketch<IntegerSummary> unionAll() {
+ private TupleSketch<IntegerSummary> unionAll() {
if (_union == null) {
- _union = new Union<>(_nominalEntries, _setOperations);
+ _union = new TupleUnion<>(_nominalEntries, _setOperations);
}
// Return the default update "gadget" sketch as a compact sketch
if (isEmpty()) {
@@ -83,7 +83,7 @@ public class TupleIntSketchAccumulator extends CustomObjectAccumulator<Sketch<In
}
// Performance optimization: ensure that the minimum Theta is used for "early stop".
- // The "early stop" optimization is implemented in the Apache Datasketches Union operation for
+ // The "early stop" optimization is implemented in the Apache Datasketches TupleUnion operation for
// ordered and compact Theta sketches. Internally, a compact and ordered Theta sketch can be
// compared to a sorted array of K items. When performing a union, only those items from
// the input sketch less than Theta need to be processed. The loop terminates as soon as a hash
@@ -91,8 +91,8 @@ public class TupleIntSketchAccumulator extends CustomObjectAccumulator<Sketch<In
// The following "sort" improves on this further by selecting the minimal Theta value up-front,
// which results in fewer redundant entries being retained and subsequently discarded during the
// union operation.
- _accumulator.sort(Comparator.comparingDouble(Sketch::getTheta));
- for (Sketch<IntegerSummary> accumulatedSketch : _accumulator) {
+ _accumulator.sort(Comparator.comparingDouble(TupleSketch::getTheta));
+ for (TupleSketch<IntegerSummary> accumulatedSketch : _accumulator) {
_union.union(accumulatedSketch);
}
_accumulator.clear();
diff --git a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/CustomSerDeUtils.java b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/CustomSerDeUtils.java
index 2879c474a58..00437f1e9aa 100644
--- a/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/CustomSerDeUtils.java
+++ b/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/CustomSerDeUtils.java
@@ -25,10 +25,11 @@ import com.google.common.primitives.Longs;
import com.tdunning.math.stats.MergingDigest;
import com.tdunning.math.stats.TDigest;
import java.io.IOException;
+import java.lang.foreign.MemorySegment;
import java.nio.ByteBuffer;
import org.apache.datasketches.cpc.CpcSketch;
-import org.apache.datasketches.memory.Memory;
-import org.apache.datasketches.theta.Sketch;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.tuple.TupleSketch;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.datasketches.tuple.aninteger.IntegerSummaryDeserializer;
import org.apache.pinot.segment.local.customobject.AvgPair;
@@ -239,10 +240,10 @@ public class CustomSerDeUtils {
}
};
- public static final ObjectSerDe<Sketch> DATA_SKETCH_THETA_SER_DE = new ObjectSerDe<Sketch>() {
+ public static final ObjectSerDe<ThetaSketch> DATA_SKETCH_THETA_SER_DE = new ObjectSerDe<ThetaSketch>() {
@Override
- public byte[] serialize(Sketch value) {
+ public byte[] serialize(ThetaSketch value) {
// The serializer should respect existing ordering to enable "early stop"
// optimisations on unions.
if (!value.isCompact()) {
@@ -252,36 +253,36 @@ public class CustomSerDeUtils {
}
@Override
- public Sketch deserialize(byte[] bytes) {
- return Sketch.wrap(Memory.wrap(bytes));
+ public ThetaSketch deserialize(byte[] bytes) {
+ return ThetaSketch.wrap(MemorySegment.ofArray(bytes).asReadOnly());
}
@Override
- public Sketch deserialize(ByteBuffer byteBuffer) {
+ public ThetaSketch deserialize(ByteBuffer byteBuffer) {
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes);
- return Sketch.wrap(Memory.wrap(bytes));
+ return ThetaSketch.wrap(MemorySegment.ofArray(bytes).asReadOnly());
}
};
- public static final ObjectSerDe<org.apache.datasketches.tuple.Sketch<IntegerSummary>> DATA_SKETCH_INT_TUPLE_SER_DE =
- new ObjectSerDe<org.apache.datasketches.tuple.Sketch<IntegerSummary>>() {
+ public static final ObjectSerDe<TupleSketch<IntegerSummary>> DATA_SKETCH_INT_TUPLE_SER_DE =
+ new ObjectSerDe<TupleSketch<IntegerSummary>>() {
@Override
- public byte[] serialize(org.apache.datasketches.tuple.Sketch<IntegerSummary> value) {
+ public byte[] serialize(TupleSketch<IntegerSummary> value) {
return value.compact().toByteArray();
}
@Override
- public org.apache.datasketches.tuple.Sketch<IntegerSummary> deserialize(byte[] bytes) {
- return org.apache.datasketches.tuple.Sketches.heapifySketch(Memory.wrap(bytes),
+ public TupleSketch<IntegerSummary> deserialize(byte[] bytes) {
+ return TupleSketch.heapifySketch(MemorySegment.ofArray(bytes),
new IntegerSummaryDeserializer());
}
@Override
- public org.apache.datasketches.tuple.Sketch<IntegerSummary> deserialize(ByteBuffer byteBuffer) {
+ public TupleSketch<IntegerSummary> deserialize(ByteBuffer byteBuffer) {
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes);
- return org.apache.datasketches.tuple.Sketches.heapifySketch(Memory.wrap(bytes),
+ return TupleSketch.heapifySketch(MemorySegment.ofArray(bytes),
new IntegerSummaryDeserializer());
}
};
@@ -295,14 +296,14 @@ public class CustomSerDeUtils {
@Override
public CpcSketch deserialize(byte[] bytes) {
- return CpcSketch.heapify(Memory.wrap(bytes));
+ return CpcSketch.heapify(MemorySegment.ofArray(bytes));
}
@Override
public CpcSketch deserialize(ByteBuffer byteBuffer) {
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes);
- return CpcSketch.heapify(Memory.wrap(bytes));
+ return CpcSketch.heapify(MemorySegment.ofArray(bytes));
}
};
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/aggregator/DistinctCountThetaSketchValueAggregatorTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/aggregator/DistinctCountThetaSketchValueAggregatorTest.java
index f368ac11055..349750298e7 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/aggregator/DistinctCountThetaSketchValueAggregatorTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/aggregator/DistinctCountThetaSketchValueAggregatorTest.java
@@ -20,10 +20,9 @@ package org.apache.pinot.segment.local.aggregator;
import java.util.List;
import java.util.stream.IntStream;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.Sketches;
-import org.apache.datasketches.theta.Union;
-import org.apache.datasketches.theta.UpdateSketch;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.ThetaUnion;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
import org.apache.pinot.spi.utils.CommonConstants;
import org.testng.annotations.Test;
@@ -43,14 +42,14 @@ public class DistinctCountThetaSketchValueAggregatorTest {
@Test
public void initialShouldParseASketch() {
- UpdateSketch input = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input = UpdatableThetaSketch.builder().build();
IntStream.range(0, 1000).forEach(input::update);
- Sketch result = input.compact();
+ ThetaSketch result = input.compact();
DistinctCountThetaSketchValueAggregator agg = new DistinctCountThetaSketchValueAggregator(List.of());
byte[] bytes = agg.serializeAggregatedValue(result);
- Sketch initSketch = toSketch(agg.getInitialAggregatedValue(bytes));
- Union union =
- Union.builder().setNominalEntries(CommonConstants.Helix.DEFAULT_THETA_SKETCH_NOMINAL_ENTRIES).buildUnion();
+ ThetaSketch initSketch = toSketch(agg.getInitialAggregatedValue(bytes));
+ ThetaUnion union =
+ ThetaUnion.builder().setNominalEntries(CommonConstants.Helix.DEFAULT_THETA_SKETCH_NOMINAL_ENTRIES).buildUnion();
union.union(initSketch);
assertEquals(initSketch.getEstimate(), result.getEstimate());
// and should update the max size
@@ -59,9 +58,9 @@ public class DistinctCountThetaSketchValueAggregatorTest {
@Test
public void initialShouldParseMultiValueSketches() {
- UpdateSketch input1 = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input1 = UpdatableThetaSketch.builder().build();
input1.update("hello");
- UpdateSketch input2 = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input2 = UpdatableThetaSketch.builder().build();
input2.update("world");
DistinctCountThetaSketchValueAggregator agg = new DistinctCountThetaSketchValueAggregator(List.of());
byte[][] bytes = {agg.serializeAggregatedValue(input1), agg.serializeAggregatedValue(input2)};
@@ -70,39 +69,39 @@ public class DistinctCountThetaSketchValueAggregatorTest {
@Test
public void applyAggregatedValueShouldUnion() {
- UpdateSketch input1 = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input1 = UpdatableThetaSketch.builder().build();
IntStream.range(0, 1000).forEach(input1::update);
- Sketch result1 = input1.compact();
- UpdateSketch input2 = Sketches.updateSketchBuilder().build();
+ ThetaSketch result1 = input1.compact();
+ UpdatableThetaSketch input2 = UpdatableThetaSketch.builder().build();
IntStream.range(0, 1000).forEach(input2::update);
- Sketch result2 = input2.compact();
+ ThetaSketch result2 = input2.compact();
DistinctCountThetaSketchValueAggregator agg = new DistinctCountThetaSketchValueAggregator(List.of());
- Sketch result = toSketch(agg.applyAggregatedValue(result1, result2));
- Union union =
- Union.builder().setNominalEntries(CommonConstants.Helix.DEFAULT_THETA_SKETCH_NOMINAL_ENTRIES).buildUnion();
+ ThetaSketch result = toSketch(agg.applyAggregatedValue(result1, result2));
+ ThetaUnion union =
+ ThetaUnion.builder().setNominalEntries(CommonConstants.Helix.DEFAULT_THETA_SKETCH_NOMINAL_ENTRIES).buildUnion();
union.union(result1);
union.union(result2);
- Sketch merged = union.getResult();
+ ThetaSketch merged = union.getResult();
assertEquals(result.getEstimate(), merged.getEstimate());
assertEquals(agg.getMaxAggregatedValueByteSize(), union.getCurrentBytes());
}
@Test
public void applyRawValueShouldUnion() {
- UpdateSketch input1 = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input1 = UpdatableThetaSketch.builder().build();
IntStream.range(0, 1000).forEach(input1::update);
- Sketch result1 = input1.compact();
- UpdateSketch input2 = Sketches.updateSketchBuilder().build();
+ ThetaSketch result1 = input1.compact();
+ UpdatableThetaSketch input2 = UpdatableThetaSketch.builder().build();
IntStream.range(0, 1000).forEach(input2::update);
- Sketch result2 = input2.compact();
+ ThetaSketch result2 = input2.compact();
DistinctCountThetaSketchValueAggregator agg = new DistinctCountThetaSketchValueAggregator(List.of());
byte[] result2bytes = agg.serializeAggregatedValue(result2);
- Sketch result = toSketch(agg.applyRawValue(result1, result2bytes));
- Union union =
- Union.builder().setNominalEntries(CommonConstants.Helix.DEFAULT_THETA_SKETCH_NOMINAL_ENTRIES).buildUnion();
+ ThetaSketch result = toSketch(agg.applyRawValue(result1, result2bytes));
+ ThetaUnion union =
+ ThetaUnion.builder().setNominalEntries(CommonConstants.Helix.DEFAULT_THETA_SKETCH_NOMINAL_ENTRIES).buildUnion();
union.union(result1);
union.union(result2);
- Sketch merged = union.getResult();
+ ThetaSketch merged = union.getResult();
assertEquals(result.getEstimate(), merged.getEstimate());
// and should update the max size
assertEquals(agg.getMaxAggregatedValueByteSize(), union.getCurrentBytes());
@@ -110,13 +109,13 @@ public class DistinctCountThetaSketchValueAggregatorTest {
@Test
public void applyRawValueShouldAdd() {
- UpdateSketch input1 = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input1 = UpdatableThetaSketch.builder().build();
input1.update("hello".hashCode());
- Sketch result1 = input1.compact();
+ ThetaSketch result1 = input1.compact();
DistinctCountThetaSketchValueAggregator agg = new DistinctCountThetaSketchValueAggregator(List.of());
- Sketch result = toSketch(agg.applyRawValue(result1, "world"));
- Union union =
- Union.builder().setNominalEntries(CommonConstants.Helix.DEFAULT_THETA_SKETCH_NOMINAL_ENTRIES).buildUnion();
+ ThetaSketch result = toSketch(agg.applyRawValue(result1, "world"));
+ ThetaUnion union =
+ ThetaUnion.builder().setNominalEntries(CommonConstants.Helix.DEFAULT_THETA_SKETCH_NOMINAL_ENTRIES).buildUnion();
union.union(result);
assertEquals(result.getEstimate(), 2.0);
// and should update the max size
@@ -125,14 +124,14 @@ public class DistinctCountThetaSketchValueAggregatorTest {
@Test
public void applyRawValueShouldSupportMultiValue() {
- UpdateSketch input1 = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input1 = UpdatableThetaSketch.builder().build();
input1.update("hello");
- Sketch result1 = input1.compact();
+ ThetaSketch result1 = input1.compact();
DistinctCountThetaSketchValueAggregator agg = new DistinctCountThetaSketchValueAggregator(List.of());
String[] strings = {"hello", "world", "this", "is", "some", "strings"};
- Sketch result = toSketch(agg.applyRawValue(result1, (Object) strings));
- Union union =
- Union.builder().setNominalEntries(CommonConstants.Helix.DEFAULT_THETA_SKETCH_NOMINAL_ENTRIES).buildUnion();
+ ThetaSketch result = toSketch(agg.applyRawValue(result1, (Object) strings));
+ ThetaUnion union =
+ ThetaUnion.builder().setNominalEntries(CommonConstants.Helix.DEFAULT_THETA_SKETCH_NOMINAL_ENTRIES).buildUnion();
union.union(result);
assertEquals(result.getEstimate(), 6.0);
// and should update the max size
@@ -168,20 +167,20 @@ public class DistinctCountThetaSketchValueAggregatorTest {
@Test
public void shouldRetainSketchOrdering() {
- UpdateSketch input = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input = UpdatableThetaSketch.builder().build();
IntStream.range(0, 10).forEach(input::update);
- Sketch unordered = input.compact(false, null);
- Sketch ordered = input.compact(true, null);
+ ThetaSketch unordered = input.compact(false, null);
+ ThetaSketch ordered = input.compact(true, null);
DistinctCountThetaSketchValueAggregator agg = new DistinctCountThetaSketchValueAggregator(List.of());
assertTrue(toSketch(agg.cloneAggregatedValue(ordered)).isOrdered());
assertFalse(toSketch(agg.cloneAggregatedValue(unordered)).isOrdered());
}
- private Sketch toSketch(Object value) {
- if (value instanceof Union) {
- return ((Union) value).getResult();
- } else if (value instanceof Sketch) {
- return (Sketch) value;
+ private ThetaSketch toSketch(Object value) {
+ if (value instanceof ThetaUnion) {
+ return ((ThetaUnion) value).getResult();
+ } else if (value instanceof ThetaSketch) {
+ return (ThetaSketch) value;
} else {
throw new IllegalStateException(
"Unsupported data type for Theta Sketch aggregation: " + value.getClass().getSimpleName());
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/aggregator/IntegerTupleSketchValueAggregatorTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/aggregator/IntegerTupleSketchValueAggregatorTest.java
index fc115f40fdc..4ea2d49fab5 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/aggregator/IntegerTupleSketchValueAggregatorTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/aggregator/IntegerTupleSketchValueAggregatorTest.java
@@ -19,10 +19,10 @@
package org.apache.pinot.segment.local.aggregator;
import java.util.List;
-import org.apache.datasketches.tuple.Sketch;
-import org.apache.datasketches.tuple.Union;
-import org.apache.datasketches.tuple.aninteger.IntegerSketch;
+import org.apache.datasketches.tuple.TupleSketch;
+import org.apache.datasketches.tuple.TupleUnion;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
+import org.apache.datasketches.tuple.aninteger.IntegerTupleSketch;
import org.testng.annotations.Test;
import static org.testng.Assert.assertEquals;
@@ -31,7 +31,7 @@ import static org.testng.Assert.assertEquals;
public class IntegerTupleSketchValueAggregatorTest {
private byte[] sketchContaining(String key, int value) {
- IntegerSketch is = new IntegerSketch(16, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch is = new IntegerTupleSketch(16, IntegerSummary.Mode.Sum);
is.update(key, value);
return is.compact().toByteArray();
}
@@ -52,36 +52,36 @@ public class IntegerTupleSketchValueAggregatorTest {
@Test
public void applyAggregatedValueShouldUnion() {
- IntegerSketch s1 = new IntegerSketch(16, IntegerSummary.Mode.Sum);
- IntegerSketch s2 = new IntegerSketch(16, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch s1 = new IntegerTupleSketch(16, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch s2 = new IntegerTupleSketch(16, IntegerSummary.Mode.Sum);
s1.update("a", 1);
s2.update("b", 1);
IntegerTupleSketchValueAggregator agg =
new IntegerTupleSketchValueAggregator(List.of(), IntegerSummary.Mode.Sum);
- Sketch<IntegerSummary> merged = toSketch(agg.applyAggregatedValue(s1, s2));
+ TupleSketch<IntegerSummary> merged = toSketch(agg.applyAggregatedValue(s1, s2));
assertEquals(merged.getEstimate(), 2.0);
assertEquals(agg.getMaxAggregatedValueByteSize(), 196632);
}
@Test
public void applyRawValueShouldUnion() {
- IntegerSketch s1 = new IntegerSketch(16, IntegerSummary.Mode.Sum);
- IntegerSketch s2 = new IntegerSketch(16, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch s1 = new IntegerTupleSketch(16, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch s2 = new IntegerTupleSketch(16, IntegerSummary.Mode.Sum);
s1.update("a", 1);
s2.update("b", 1);
IntegerTupleSketchValueAggregator agg =
new IntegerTupleSketchValueAggregator(List.of(), IntegerSummary.Mode.Sum);
- Sketch<IntegerSummary> merged = toSketch(agg.applyRawValue(s1, agg.serializeAggregatedValue(s2)));
+ TupleSketch<IntegerSummary> merged = toSketch(agg.applyRawValue(s1, agg.serializeAggregatedValue(s2)));
assertEquals(merged.getEstimate(), 2.0);
assertEquals(agg.getMaxAggregatedValueByteSize(), 196632);
}
@SuppressWarnings("unchecked")
- private Sketch<IntegerSummary> toSketch(Object value) {
- if (value instanceof Union) {
- return ((Union) value).getResult();
- } else if (value instanceof Sketch) {
- return ((Sketch) value);
+ private TupleSketch<IntegerSummary> toSketch(Object value) {
+ if (value instanceof TupleUnion) {
+ return ((TupleUnion) value).getResult();
+ } else if (value instanceof TupleSketch) {
+ return ((TupleSketch) value);
} else {
throw new IllegalStateException(
"Unsupported data type for Integer Tuple Sketch aggregation: " + value.getClass().getSimpleName());
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/customobject/ThetaSketchAccumulatorTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/customobject/ThetaSketchAccumulatorTest.java
index f705e9e45b9..c9e05c51d70 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/customobject/ThetaSketchAccumulatorTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/customobject/ThetaSketchAccumulatorTest.java
@@ -20,21 +20,20 @@
package org.apache.pinot.segment.local.customobject;
import java.util.stream.IntStream;
-import org.apache.datasketches.theta.SetOperationBuilder;
-import org.apache.datasketches.theta.Sketch;
-import org.apache.datasketches.theta.Sketches;
-import org.apache.datasketches.theta.UpdateSketch;
+import org.apache.datasketches.theta.ThetaSetOperationBuilder;
+import org.apache.datasketches.theta.ThetaSketch;
+import org.apache.datasketches.theta.UpdatableThetaSketch;
import org.testng.Assert;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class ThetaSketchAccumulatorTest {
- private SetOperationBuilder _setOperationBuilder;
+ private ThetaSetOperationBuilder _setOperationBuilder;
@BeforeMethod
public void setUp() {
- _setOperationBuilder = new SetOperationBuilder();
+ _setOperationBuilder = new ThetaSetOperationBuilder();
}
@Test
@@ -46,9 +45,9 @@ public class ThetaSketchAccumulatorTest {
@Test
public void testAccumulatorWithSingleSketch() {
- UpdateSketch input = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input = UpdatableThetaSketch.builder().build();
IntStream.range(0, 1000).forEach(input::update);
- Sketch sketch = input.compact();
+ ThetaSketch sketch = input.compact();
ThetaSketchAccumulator accumulator = new ThetaSketchAccumulator(_setOperationBuilder, 2);
accumulator.apply(sketch);
@@ -59,12 +58,12 @@ public class ThetaSketchAccumulatorTest {
@Test
public void testAccumulatorMerge() {
- UpdateSketch input1 = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input1 = UpdatableThetaSketch.builder().build();
IntStream.range(0, 1000).forEach(input1::update);
- Sketch sketch1 = input1.compact();
- UpdateSketch input2 = Sketches.updateSketchBuilder().build();
+ ThetaSketch sketch1 = input1.compact();
+ UpdatableThetaSketch input2 = UpdatableThetaSketch.builder().build();
IntStream.range(1000, 2000).forEach(input2::update);
- Sketch sketch2 = input2.compact();
+ ThetaSketch sketch2 = input2.compact();
ThetaSketchAccumulator accumulator1 = new ThetaSketchAccumulator(_setOperationBuilder, 3);
accumulator1.apply(sketch1);
@@ -77,12 +76,12 @@ public class ThetaSketchAccumulatorTest {
@Test
public void testThresholdBehavior() {
- UpdateSketch input1 = Sketches.updateSketchBuilder().build();
+ UpdatableThetaSketch input1 = UpdatableThetaSketch.builder().build();
IntStream.range(0, 1000).forEach(input1::update);
- Sketch sketch1 = input1.compact();
- UpdateSketch input2 = Sketches.updateSketchBuilder().build();
+ ThetaSketch sketch1 = input1.compact();
+ UpdatableThetaSketch input2 = UpdatableThetaSketch.builder().build();
IntStream.range(1000, 2000).forEach(input2::update);
- Sketch sketch2 = input2.compact();
+ ThetaSketch sketch2 = input2.compact();
ThetaSketchAccumulator accumulator = new ThetaSketchAccumulator(_setOperationBuilder, 3);
accumulator.apply(sketch1);
diff --git a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/customobject/TupleIntSketchAccumulatorTest.java b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/customobject/TupleIntSketchAccumulatorTest.java
index 77554162115..fd7761b6c86 100644
--- a/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/customobject/TupleIntSketchAccumulatorTest.java
+++ b/pinot-segment-local/src/test/java/org/apache/pinot/segment/local/customobject/TupleIntSketchAccumulatorTest.java
@@ -20,10 +20,10 @@
package org.apache.pinot.segment.local.customobject;
import java.util.stream.IntStream;
-import org.apache.datasketches.tuple.CompactSketch;
-import org.apache.datasketches.tuple.aninteger.IntegerSketch;
+import org.apache.datasketches.tuple.CompactTupleSketch;
import org.apache.datasketches.tuple.aninteger.IntegerSummary;
import org.apache.datasketches.tuple.aninteger.IntegerSummarySetOperations;
+import org.apache.datasketches.tuple.aninteger.IntegerTupleSketch;
import org.testng.Assert;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
@@ -48,9 +48,9 @@ public class TupleIntSketchAccumulatorTest {
@Test
public void testAccumulatorWithSingleSketch() {
- IntegerSketch input = new IntegerSketch(_lgK, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch input = new IntegerTupleSketch(_lgK, IntegerSummary.Mode.Sum);
IntStream.range(0, 1000).forEach(i -> input.update(i, 1));
- CompactSketch<IntegerSummary> sketch = input.compact();
+ CompactTupleSketch<IntegerSummary> sketch = input.compact();
TupleIntSketchAccumulator accumulator = new TupleIntSketchAccumulator(_setOps, _nominalEntries, 2);
accumulator.apply(sketch);
@@ -61,12 +61,12 @@ public class TupleIntSketchAccumulatorTest {
@Test
public void testAccumulatorMerge() {
- IntegerSketch input1 = new IntegerSketch(_lgK, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch input1 = new IntegerTupleSketch(_lgK, IntegerSummary.Mode.Sum);
IntStream.range(0, 1000).forEach(i -> input1.update(i, 1));
- CompactSketch<IntegerSummary> sketch1 = input1.compact();
- IntegerSketch input2 = new IntegerSketch(_lgK, IntegerSummary.Mode.Sum);
+ CompactTupleSketch<IntegerSummary> sketch1 = input1.compact();
+ IntegerTupleSketch input2 = new IntegerTupleSketch(_lgK, IntegerSummary.Mode.Sum);
IntStream.range(1000, 2000).forEach(i -> input2.update(i, 1));
- CompactSketch<IntegerSummary> sketch2 = input2.compact();
+ CompactTupleSketch<IntegerSummary> sketch2 = input2.compact();
TupleIntSketchAccumulator accumulator1 = new TupleIntSketchAccumulator(_setOps, _nominalEntries, 3);
accumulator1.apply(sketch1);
@@ -79,12 +79,12 @@ public class TupleIntSketchAccumulatorTest {
@Test
public void testThresholdBehavior() {
- IntegerSketch input1 = new IntegerSketch(_lgK, IntegerSummary.Mode.Sum);
+ IntegerTupleSketch input1 = new IntegerTupleSketch(_lgK, IntegerSummary.Mode.Sum);
IntStream.range(0, 1000).forEach(i -> input1.update(i, 1));
- CompactSketch<IntegerSummary> sketch1 = input1.compact();
- IntegerSketch input2 = new IntegerSketch(_lgK, IntegerSummary.Mode.Sum);
+ CompactTupleSketch<IntegerSummary> sketch1 = input1.compact();
+ IntegerTupleSketch input2 = new IntegerTupleSketch(_lgK, IntegerSummary.Mode.Sum);
IntStream.range(1000, 2000).forEach(i -> input2.update(i, 1));
- CompactSketch<IntegerSummary> sketch2 = input2.compact();
+ CompactTupleSketch<IntegerSummary> sketch2 = input2.compact();
TupleIntSketchAccumulator accumulator = new TupleIntSketchAccumulator(_setOps, _nominalEntries, 3);
accumulator.apply(sketch1);
diff --git a/pinot-server/src/test/java/org/apache/pinot/server/predownload/PredownloadPredownloadStatusRecorderTest.java b/pinot-server/src/test/java/org/apache/pinot/server/predownload/PredownloadPredownloadStatusRecorderTest.java
index cd816e3fd8e..4bfc7e7a1b4 100644
--- a/pinot-server/src/test/java/org/apache/pinot/server/predownload/PredownloadPredownloadStatusRecorderTest.java
+++ b/pinot-server/src/test/java/org/apache/pinot/server/predownload/PredownloadPredownloadStatusRecorderTest.java
@@ -67,7 +67,6 @@ public class PredownloadPredownloadStatusRecorderTest {
testFolder.mkdir();
PredownloadStatusRecorder.setStatusRecordFolder(testFolder.getAbsolutePath());
- SecurityManager originalSecurityManager = System.getSecurityManager();
try {
ExitHelper.setExitAction(status -> {
throw new ExitException(status);
@@ -76,8 +75,7 @@ public class PredownloadPredownloadStatusRecorderTest {
tryRetriableFailure(testFolder);
tryNonRetriableFailure(testFolder);
} finally {
- // Restore the original SecurityManager
- System.setSecurityManager(originalSecurityManager);
+ ExitHelper.resetExitAction();
}
}
diff --git a/pom.xml b/pom.xml
index a1481aa9a80..695c3e942a5 100644
--- a/pom.xml
+++ b/pom.xml
@@ -115,12 +115,12 @@
<pinot.root>${basedir}</pinot.root>
<build.profile.id>dev</build.profile.id>
<!--
- Pinot services build against Java 21 by default. SPI/client modules consumed
+ Pinot services build against Java 25 by default. SPI/client modules consumed
by external JVMs (pinot-spi, pinot-segment-spi, pinot-timeseries-spi,
pinot-common, pinot-java-client, pinot-jdbc-client) hard-code Java 11
bytecode in their own maven-compiler-plugin configuration.
-->
- <jdk.version>21</jdk.version>
+ <jdk.version>25</jdk.version>
<!-- OS classifier for platform-specific protoc/grpc binaries, overridden by OS detection profiles -->
<os.detected.classifier>linux-x86_64</os.detected.classifier>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
@@ -220,7 +220,7 @@
<snakeyaml.version>2.6</snakeyaml.version>
<hadoop-shaded-protobuf_3_25.version>1.5.0</hadoop-shaded-protobuf_3_25.version>
<clearspring-stream-lib.version>2.9.8</clearspring-stream-lib.version>
- <datasketches-java.version>6.2.0</datasketches-java.version>
+ <datasketches-java.version>9.0.0</datasketches-java.version>
<t-digest.version>3.2</t-digest.version>
<picocli.version>4.7.7</picocli.version>
<tyrus-standalone-client.version>2.2.2</tyrus-standalone-client.version>
@@ -2459,8 +2459,8 @@
<configuration>
<rules>
<requireJavaVersion>
- <version>[21,)</version>
- <message>Apache Pinot requires JDK 21 or newer to build.</message>
+ <version>[25,)</version>
+ <message>Apache Pinot requires JDK 25 or newer to build.</message>
</requireJavaVersion>
</rules>
<fail>true</fail>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org