You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datasketches.apache.org by le...@apache.org on 2026/06/01 00:52:13 UTC
(datasketches-memory) branch update_7.0.X updated: update update_7.0.X
This is an automated email from the ASF dual-hosted git repository.
leerho pushed a commit to branch update_7.0.X
in repository https://gitbox.apache.org/repos/asf/datasketches-memory.git
The following commit(s) were added to refs/heads/update_7.0.X by this push:
new b604d73b update update_7.0.X
b604d73b is described below
commit b604d73b4b1557ae3cec24baba2cfa1b78d76393
Author: Lee Rhodes <le...@gmail.com>
AuthorDate: Sun May 31 17:52:06 2026 -0700
update update_7.0.X
---
.github/workflows/auto-jdk-os-matrix.yml | 2 +-
README.md | 278 ++++++++++++-------------------
docs/maven.md | 50 ------
3 files changed, 111 insertions(+), 219 deletions(-)
diff --git a/.github/workflows/auto-jdk-os-matrix.yml b/.github/workflows/auto-jdk-os-matrix.yml
index 20e80ce4..ffa710fd 100644
--- a/.github/workflows/auto-jdk-os-matrix.yml
+++ b/.github/workflows/auto-jdk-os-matrix.yml
@@ -1,4 +1,4 @@
-name: Auto JDK & OS Matrix Test & Install
+name: JDK & OS Matrix Test
on:
# push:
diff --git a/README.md b/README.md
index d103a97f..ce8c68c6 100644
--- a/README.md
+++ b/README.md
@@ -23,214 +23,156 @@
=================
-# DataSketches Java Memory Component
-This Memory component is general purpose, has no external runtime dependencies and can be used in any
-application that needs to manage data structures inside or outside the Java heap.
+# DataSketches Memory Project
+This Memory Project was initially developed to enable the DataSketches Java Project to manage off-heap data structures efficiently. The first version was developed as an internal project at Yahoo in 2014 using Java 8 and released to open source in 2015. This Memory Project is general purpose, has no external runtime dependencies and can be used in any application that needs to manage data structures on or off the Java heap.
-Please visit the main [DataSketches website](https://datasketches.apache.org) for more information.
+## Basic Capabilities
+* **Speed**: Fast Read & Write using Unsafe
+* **Types**: All Java primitives plus Java Strings
+* **Target**: A contiguous *region* of on-heap or off-heap memory. (Note: Java FFM later named this *region* a *MemorySegment*)
+* **Addressing**: Arbitrary Byte Offset (long) as well as relative positional offset.
+* **Size**: A region of off-heap memory may exceed the standard Java limit of 2^31-1 bytes.
-If you are interested in making contributions to this Memory component please see our
-[Community](https://datasketches.apache.org/docs/Community/) page.
+To make this useful, this project also provided:
-The goal of this component of the DataSketches library is to provide a high performance access API for accessing six different types of memory resources. Each of the six resource types is accessed using different API methods.
+## Capability Set A:
+* Wrapping of Java primitive arrays as a region.
+* Wrapping of ByteBuffers (both on and off-heap) as a region.
+* Creation and immediate closing of off-heap regions of memory.
+* Creation and immediate closing of off-heap memory-mapped regions of a file.
-Note: *primitive* = *{byte, short, int, long, float, double}*
+Providing these capabilities without sacrificing performance required leveraging internal JVM classes such as *Unsafe*, *FileChannelImpl*, *DirectBuffer*, *MappedByteBuffer internals* and *sun.nio.ch*.
-### Contiguous bytes on the Java Heap constructed by:
+However, starting with Java 11, Oracle started restricting access to a many JVM internal classes. With Java 17 the internals of *FileChannelImpl* and *MappedByteBuffer* were hidden or relocated. This blocked the ability to provide direct access to file-mapped memory.
-* **Heap:**
- * *Memory.wrap(primitive[])*
- * *WritableMemory.allocate(int)*
- * *WritableMemory.writableWrap(primitive[])*
+Meanwhile, the Java Panama project, which had the potential to provide all the above capabilities as part of the Java language, became available as the Foreign Function and Memory (FFM) API in the form of "Incubation" code in Java 17 and as "Preview" code in Java 21.
-* **Heap via ByteBuffer**
- * *Memory.wrap(ByteBuffer.allocate(int))*
- * *WritableMemory.writableWrap(ByteBuffer.allocate(int))*
+Unfortunately, the DataSketches Memory Project releases 4.X, 5.X and 6.X for Java 17 and 21 were developed using the Incubation and Preview versions of FFM not realizing that Incubation and Preview codes are only available until the next java version is released, after which they are no longer available.
-### Contiguous bytes off the Java Heap constructed by:
+The actual LTS release of FFM did not occur until Java 25 was released on September 16, 2025. Since Java 17 was released on September 14, 2021, there is a gap of 4 years where Oracle had blocked access to internals of the JVM without providing a LTS release API with comparable capabilities. This means that for the LTS Java versions 17 and 21, the DataSketches Memory Project could not provide access to file-mapped memory. Nonetheless, it is still possible to provide these capabilities:
-* **Direct:**
- * *WritableMemory.allocateDirect(long)* method.
+## Capability Set B:
+* Wrapping of Java primitive arrays as a region.
+* Wrapping of ByteBuffers (both on and off-heap) as a region.
+* Creation and immediate closing of off-heap regions of memory.
-* **Direct via ByteBuffer**
- * *WritableMemory.writableWrap(ByteBuffer.allocateDirect(int))*
+The objective of this 7.X release of the Memory Project is to provide **Capability Set A** with Java 11 and **Capability Set B** for Java versions 17 and 21 for users that are still dependent on the these older LTS releases.
-* **Memory-Mapped Files**
- * *WritableMemory.writableMap(File)* method.
+In addition, with this release the build code has been completely refactored:
-# Releases 3.0.0 (inclusive) to 4.0.0 (exclusive)
-These are transitional releases that also supports Java 8 and 11.
-However, the goal of this set of releases is to migrate the API to what it will be in release 4.0.0.
-The 4.0.0 release will require Java 17 and will utilize the Project Panama (FFM) capabilites introduced in Java 17.
+* A clean all-Maven build, install and deploy process ( no shell scripts required ).
+* New GitHub Actions Workflows that test this project in matrix of 3 OSs (Windows, MacOS, Ubuntu) and 3 Java versions (11, 17, 21).
+* All code jar files, including the test jars, can be successfully run as jars.
+* The Apache release source is a multi-module Maven project
+* The the release to Maven Central is two jar sets as two new artifacts:
+ * datasketches-memory-java11
+ * datasketches-memory-java17_21
+* All documentation has been brought up-to-date.
-Some of the capabilites of releases [2.0.0, 3.0.0) have been removed because they cannot be supported in release 4.0.0 with Panama-17.
-For example:
+Because the FFM capabilities built into Java 25 (released September, 2025) are a major superset of this Memory Project's capabilities (open sourced in 2015), all of these capabilities are now available directly from the Java Language. There is little reason to continue to provide new releases of this project. Alas, this project must come to an end. Any further releases will only be for bug fixes. Nonetheless, we were able to provide these fast, low-level, off-heap memory capabilities for [...]
-* The ability to directly import and export UTF-8 encoded strings has been removed.
- * The UTF-8 code in Memory 1.0.0 through 2.2.1 is out of date and needed to be removed anyway.
- * Java already has built-in UTF-8 encoding.
-* The ability to directly import and export boolean arrays has been removed.
- * The Java Language Specification does not define the byte storage format for boolean arrays, thus Panama-17 doesn't support boolean arrays.
-* Other minor changes to the API is best understood by examining the Javadocs directly.
+## Comments about ByteBuffer
+Although the ByteBuffer provides similar capabilities, it has some severe limitations.
-Release 3.0.0 includes two bug fixes (Issues #194, #195).
+* The deallocation is governed by the Garbage Collector. In large systems this delay can lead to out-of-memory failures.
+* The addressing is limited to 2^31 -1 elements.
+* Reading and writing multi-byte primitive arrays require recasting the ByteBuffer to a <*type*>Buffer where the alignment becomes modulo the *type*. This severely complicates the creation of packed structs.
+* The API has some serious footguns, which can catch even experienced programmers and can be challenging to debug:
+ * Every duplicate() or slice() resets the endianness to Big Endian.
+ * Numerous common operations silently discard the *mark*.
-It is our expectation that this set of releases will be the last that support Java 8 and 11.
+----
-The comments in the following section for releases [2.0.0, 3.0.0) still apply.
+## Release Comments
-# Releases 2.0.0 (inclusive) to 3.0.0 (exclusive)
-Starting with release *datasketches-memory-2.0.0*, this Memory component supports Java 8 and 11. Providing access to the off-heap resources in Java 8 only requires reflection. However, **Java 9 introduced the Java Platform Module System (JPMS) where access to these internal classes requires starting up the JVM with special JPMS arguments.** The actual JVM arguments required will depend on how the user intends to use the Memory API, the Java version used to run the user's application and [...]
+### For Use With Java 8
+Please download datasketches-memory 3.X from the website [downloads](https://datasketches.apache.org/docs/Community/Downloads.html) page and refer to the README.md there.
+The 3.X set of releases will be the last that support Java 8.
-Also see the [usage examples](docs/usage-examples.md) for more information.
+### Releases 4.X, 5.X and 6.X
+Deprecated with the 7.X releases.
-## USE AS A LIBRARY (using jars from Maven Central)
-In this environment, the user is using the Jars from Maven Central as a library dependency and not attempting to build the Memory component from the source code or run the Memory component tests.
+### Releases 7.X (THIS RELEASE)
+* This release no longer uses JPMS but the jar manifests do provide an Automatic Module Name.
+* In addition to the source code obtained from Apache, this release provides two sets of jars in Maven Central based on two artifacts:
-* If you are running Java 8, no extra JVM arguments are required.
-* If you are running Java 11-13 and only using the **Heap** related API, no extra JVM arguments are required.
+#### Artifact: datasketches-memory-java11 (Capability Set A)
+This artifact only supports Java 11
-Otherwise, if you are running Java 11-13 and ...
+#### Artifact: datasketches-memory-java17_21 (Capability Set B)
+This artifact only supports Java 17 and Java 21
-* If your application **is not a JPMS module** use the following table. Choose the columns that describe your use of the Memory API. If any of the columns contain a *Yes*, then the JVM argument in the first column of the row containing a *Yes* will be required. If you are not sure the extent of the Memory API being used, there is no harm in specifying all 4 JVM arguments. Note: do not embed any spaces in the full argument.
+----
-| JVM Arguments for non-JPMS Applications | Direct ByteBuffer | Direct | MemoryMapped Files |
-| :----------------------------------------------------: | :---------------: | :----: | :----------------: |
-| --add-exports java.base/jdk.internal.misc= ALL-UNNAMED | | Yes | |
-| --add-exports java.base/jdk.internal.ref= ALL-UNNAMED | | Yes | Yes |
-| --add-opens java.base/java.nio= ALL-UNNAMED | | Yes | Yes |
-| --add-opens java.base/sun.nio.ch= ALL-UNNAMED | | | Yes |
+## LIBRARY USAGE (using jars from Maven Central)
+In this environment, the user is using the Jars from Maven Central as a library dependency and not attempting to build this Memory Project from the source code.
-* If your application **is a JPMS module** use the following table. Choose the columns that describe your use of the Memory API. If any of the columns contain a *Yes*, then the JVM argument in the first column of the row containing a *Yes* will be required. If you are not sure the extent of the Memory API being used, there is no harm in specifying all 4 JVM arguments. Note: do not embed any spaces in the full argument.
-| JVM Arguments for JPMS Applications | Direct ByteBuffer | Direct | MemoryMapped Files |
-| :-----------------------------------------------------------------------: | :---------------: | :----: | :----------------: |
-| --add-exports java.base/jdk.internal.misc= org.apache.datasketches.memory | | Yes | |
-| --add-exports java.base/jdk.internal.ref= org.apache.datasketches.memory | | Yes | Yes |
-| --add-opens java.base/java.nio= org.apache.datasketches.memory | | Yes | Yes |
-| --add-opens java.base/sun.nio.ch= org.apache.datasketches.memory | Yes | | Yes |
+### If you are running Java 11 you must use artifact *datasketches-memory-java11*
+You will need to add these java modular arguments to your JVM:
+* --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
+* --add-exports=java.base/sun.nio.ch=ALL-UNNAMED
+* --add-opens=java.base/java.nio=ALL-UNNAMED
+* --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
-## DEVELOPER USAGE
-In this environment the developer needs to build the Memory component from source and run the Memory Component tests. There are two use cases. The first is for a *System Developer* that needs to build and test their own Jar from source for a specific Java version. The second use case is for a *Memory Component Developer and Contributor*.
+### If you are running Java 17 or Java 21 you must use artifact *datasketches-memory-java17_21*
+You will need to add these java modular arguments to your JVM:
-* System Developer
- * Compile, test and create a Jar for a specific Java version
- * use the provided script for this purpose
+* --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED
+* --add-exports=java.base/sun.nio.ch=ALL-UNNAMED
-* Memory Component Developer / Contributor
- * Compile & test the library from source code using:
- * Eclipse (version)
- * IntelliJ (version)
- * Maven (version)
- * Command-line or scripts
- * The developer must have installed in their development system at least JDK versions 8 and 11.
- * Unless building with the provided script, the developer must have a valid [Maven toolchain configuration](docs/maven-toolchains.md).
+## DEVELOPER USAGE
+In this environment the developer needs to build the Memory component from source and run the Memory Component tests.
+* Fork this project at https://github.com/apache/datasketches-memory into your own GitHub development site. The 7.0.X releases will be located in the branch 7.0.X.
+* Provide a [toolchains.xml](https://maven.apache.org/guides/mini/guide-using-toolchains.html) in your local system at `~/.m2/toolchains.xml` and configured with at least JDK versions 11, 17 and 21.
+* You must have gpg-agent running in your terminal.
+* Your local machine JAVA_HOME should be set to at least Java 8, but Java 17 or higher is recommended.
-### Build Instructions
+### Maven Build Instructions
__NOTES:__
-1) This component accesses resource files for testing. As a result, the directory elements
- of the full absolute path of the target installation directory must qualify as Java identifiers.
- In other words, the directory elements must not have any space characters (or non-Java identifier
- characters) in any of the path elements. This is required by the Oracle Java Specification in
- order to ensure location-independent access to resources:
- [See Oracle Location-Independent Access to Resources](https://docs.oracle.com/javase/8/docs/technotes/guides/lang/resources.html)
-
-#### Dependencies
-There are no run-time dependencies. See the pom.xml file for test dependencies.
-
-#### Maven build instructions
-The Maven build requires the following JDKs to compile:
+* This component accesses resource files for testing. As a result, the directory elements of the full absolute path of the target installation directory must qualify as Java identifiers. In other words, the directory elements must not have any space characters (or non-Java identifier characters) in any of the path elements. This is required by the Oracle Java Specification in order to ensure location-independent access to resources: [See Oracle Location-Independent Access to Resources](h [...]
-- JDK8/Hotspot
-- JDK11/Hotspot
+__TESTS:__
-Before building, first ensure that your local environment has been configured according to the [Maven Toolchains Configuration](docs/maven-toolchains.md).
+* There are two types of tests: unit tests and continuous integration (CI) tests.
+The CI tests target the released jars and run all the unit tests against Java versions 11, 17 and 21 and three different OSs, Windows, MacOS and Ubuntu.
-There are two types of tests: normal unit tests and continuous integration(CI) tests.
-The CI tests target the Multi-Release (MR) JAR and run the entire test suite using a specific version of Java. Running the CI test command also runs the default unit tests.
-
-To run normal unit tests:
-
- mvn clean test
+To run normal unit tests on the forked site:
+* mvn clean test
+* The test results can be found in `/target/test-output`
To run javadoc on this multi-module project, use:
-
- mvn clean javadoc:javadoc -DskipTests=true
-
-To build the multi-release JAR, use:
-
- mvn clean package
-
-To run the eclipse plugin on this multi-module project, use:
-
- mvn clean eclipse:eclipse -DskipTests=true
-
-To install jars built from the downloaded source:
-
- mvn clean install -DskipTests=true
-
-This will create the following Jars:
-
-* datasketches-memory-X.Y.Z.jar The compiled main class files.
-* datasketches-memory-X.Y.Z-tests.jar The compiled test class files.
-* datasketches-memory-X.Y.Z-sources.jar The main source files.
-* datasketches-memory-X.Y.Z-test-sources.jar The test source files
-* datasketches-memory-X.Y.Z-javadoc.jar The compressed Javadocs.
-
-#### Building for a specific java version
-
-A build script named **package-single-release-jar.sh** has been provided to package a JAR for a specific java version. This is necessary in cases where a developer is unable to install all the required versions of the JDK that are required as part of the Maven build.
-
-The build script performs the following steps:
-
-1. Sets up staging directories under **target/** for the package files
-2. Uses git commands to gather information about the current Git commit and branch
-3. Compiles java source tree
-4. Packages a JAR containing compiled sources together with the Manifest, License and Notice files
-5. Checks and tests the assembled JAR by using the API to access four different resource types
-
-The build script is located in the **tools/scripts/** directory and requires the following arguments:
-
-* JDK Home Directory - The first argument is the absolute path of JDK home directory e.g. $JAVA_HOME
-* Git Version Tag - The second argument is the Git Version Tag for this deployment e.g. 1.0.0-SNAPSHOT, 1.0.0-RC1, 1.0.0 etc.
-* Project Directory - The third argument is the absolute path of project.basedir e.g. /src/apache-datasketches-memory
-
-For example, if the project base directory is `/src/datasketches-memory`;
-
-To run the script for a release version:
-
- ./tools/scripts/package-single-release-jar.sh $JAVA_HOME 2.1.0 /src/datasketches-memory
-
-To run the script for a snapshot version:
-
- ./tools/scripts/package-single-release-jar.sh $JAVA_HOME 2.2.0-SNAPSHOT /src/datasketches-memory
-
-To run the script for an RC version:
-
- ./tools/scripts/package-single-release-jar.sh $JAVA_HOME 2.1.0-RC1 /src/datasketches-memory
-
-Note that the script does **not** use the _Git Version Tag_ to adjust the working copy to a remote tag - it is expected that the user has a pristine copy of the desired branch/tag available **before** using the script.
+* mvn clean javadoc:javadoc -DskipTests=true
+* The javadocs will be located in `/target/site/apidocs/`
+
+To install the jars in ~/.m2
+* mvn clean install -DskipTests=true
+
+This will create the following sets of Jars and POM, with associated GPG .asc signatures in the folder `~/.m2/repository/org/apache/datasketches/` and under
+
+`datasketches-memory-java11/X.Y.Z/`:
+* datasketches-memory-java11-X.Y.Z.jar
+* datasketches-memory-java11-X.Y.Z-tests.jar
+* datasketches-memory-java11-X.Y.Z-sources.jar
+* datasketches-memory-java11-X.Y.Z-test-sources.jar
+* datasketches-memory-java11-X.Y.Z-javadoc.jar
+* datasketches-memory-java11-X.Y.Z-pom
+
+And under:
+`datasketches-memory-java17_21/X.Y.Z/`:
+* datasketches-memory-java17_21-X.Y.Z.jar
+* datasketches-memory-java17_21-X.Y.Z-tests.jar
+* datasketches-memory-java17_21-X.Y.Z-sources.jar
+* datasketches-memory-java17_21-X.Y.Z-test-sources.jar
+* datasketches-memory-java17_21-X.Y.Z-javadoc.jar
+* datasketches-memory-java17_21-X.Y.Z-pom
---
-### Further documentation for contributors
-
-For more information on the project configuration, the following topics are discussed in more
-detail:
+Please visit the main [DataSketches website](https://datasketches.apache.org) for more information about the DataSketches Library and all the different languages it is available in.
-* [Maven Configuration](docs/maven.md)
-* [Maven Toolchains Configuration](docs/maven-toolchains.md)
-* [Multi-Release Jar](docs/multi-release-jar.md)
-* [Java Platform Module System](docs/module-system.md)
-* [Usage examples](docs/usage-examples.md)
-
-In order to build and contribute to this project, please read the relevant IDE documentation:
-
-- [Eclipse IDE Setup](docs/eclipse.md)
-- [IntelliJ IDE Setup](docs/intellij.md)
-
-For releasing to AppNexus, please use the `sign-deploy-jar.sh` script in the scripts directory.
-See the documentation within the script for usage instructions.
+If you are interested in making contributions to this Memory Project or to any of the other DataSketches Projects please visit our
+[Community](https://datasketches.apache.org/docs/Community/) page.
diff --git a/docs/maven.md b/docs/maven.md
deleted file mode 100644
index 5fd9868f..00000000
--- a/docs/maven.md
+++ /dev/null
@@ -1,50 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
--->
-
-# Maven Configuration: Multi-Module Project
-
-This project is a multi-module Maven project. A multi-module Maven project consists of an
-aggregator project (the `datasketches-memory` project), together with a set of submodules.
-The aggregator's configuration is inherited by each submodule, thus reducing duplication.
-
-Datasketches Memory makes use of some features of the Java platform, for example, `Unsafe`,
-which have evolved in Java versions 9 and above. Therefore, a multi-module project allows us to
-add support for later versions of Java by using independent Maven modules to target
-platform-specific APIs. For example, to deallocate references a `sun.misc.Cleaner` will be used
-in Java8, but the `jdk.internal.ref.Cleaner` is used in Java 11.
-
-This project has been divided into the following submodules:
-
-* datasketches-memory-java8 (base version of the JVM that is currently supported)
-* datasketches-memory-java11 (Java11 equivalent of some platform specific classes in
-
-### Artifact assembly
-
-During the Maven package phase, the [Maven exec plugin](https://www.mojohaus.org/exec-maven-plugin/index.html)
-invokes a custom bash script `package-mr-jar.sh` located in the `tools/scripts` directory.
-This script assembles sources and compiled classes from the individual maven modules into
-a coherent set of JARs that appear to the end user to originate from a single module.
-
-The following jars are assembled by the bash package script:
-
-* datasketches-memory-X.Y.Z.jar The compiled main class files.
-* datasketches-memory-X.Y.Z-tests.jar The compiled test class files.
-* datasketches-memory-X.Y.Z-sources.jar The main source files.
-* datasketches-memory-X.Y.Z-test-sources.jar The test source files
-* datasketches-memory-X.Y.Z-javadoc.jar The compressed Javadocs.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datasketches.apache.org
For additional commands, e-mail: commits-help@datasketches.apache.org