You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@baremaps.apache.org by bc...@apache.org on 2023/09/21 21:08:16 UTC
[incubator-baremaps] branch pmtiles updated (60ce43ef -> f65b5eeb)
This is an automated email from the ASF dual-hosted git repository.
bchapuis pushed a change to branch pmtiles
in repository https://gitbox.apache.org/repos/asf/incubator-baremaps.git
discard 60ce43ef Refactor package and introduce a PMTilesReader class
discard 18972410 Fix unit tests
discard 8b24b808 Port additional methods and tests
discard 1856fd16 Format code
discard ad8bd43e Implement header serialization and deserialization
discard 97ac1b61 Add pmtiles functions
add 49a84c96 Expose environment variables to javascript (#776)
add dfa1e949 Add a datasource object to the tileset (#777)
add 78397fc6 List the derived files in the NOTICE (#771)
add 894e959a Concat THIRD-PARTY to LICENSE in bin release (#778)
new a30241e2 Add pmtiles functions
new 4c91cc6f Implement header serialization and deserialization
new 61e28f36 Format code
new 5ef6a50f Port additional methods and tests
new dbae519e Fix unit tests
new 39f811d5 Refactor package and introduce a PMTilesReader class
new 4633de7b Add a minimalistic tile writer
new f65b5eeb Correctly set header and metadata
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (60ce43ef)
\
N -- N -- N refs/heads/pmtiles (f65b5eeb)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
DISCLAIMER-WIP | 4 +
LICENSE | 4 +-
NOTICE | 38 +-
baremaps-cli/licenseBundledDeps.ftl | 56 ---
baremaps-cli/pom.xml | 72 ++-
baremaps-cli/src/assembly/bin.xml | 8 +-
baremaps-cli/src/license/bundle.ftl | 37 ++
baremaps-cli/src/license/override.ftl | 36 ++
baremaps-cli/src/license/override.properties | 209 ++++++++
.../main/java/org/apache/baremaps/cli/map/Dev.java | 2 +-
.../java/org/apache/baremaps/cli/map/Serve.java | 4 +-
.../org/apache/baremaps/config/ConfigReader.java | 7 +
.../org/apache/baremaps/tilestore/TileCache.java | 6 +
.../org/apache/baremaps/tilestore/TileStore.java | 2 +-
.../baremaps/tilestore/file/FileTileStore.java | 5 +
.../baremaps/tilestore/mbtiles/MBTilesStore.java | 5 +
.../baremaps/tilestore/pmtiles/Compression.java | 109 ++--
.../apache/baremaps/tilestore/pmtiles/Entry.java | 104 ++--
.../apache/baremaps/tilestore/pmtiles/Header.java | 556 ++++++++++++---------
.../apache/baremaps/tilestore/pmtiles/PMTiles.java | 263 ++++++----
.../baremaps/tilestore/pmtiles/PMTilesReader.java | 91 ++++
.../PMTilesStore.java} | 48 +-
.../baremaps/tilestore/pmtiles/PMTilesWriter.java | 124 +++++
.../baremaps/tilestore/pmtiles/TileType.java | 24 +-
.../tilestore/postgres/PostgresTileStore.java | 4 +
.../org/apache/baremaps/utils/PostgresUtils.java | 104 +++-
.../baremaps/vectortile/tileset/Database.java | 194 +++++++
.../baremaps/vectortile/tileset/Tileset.java | 6 +-
.../apache/baremaps/workflow/WorkflowContext.java | 6 +-
.../apache/baremaps/workflow/tasks/ExecuteSql.java | 2 +-
.../baremaps/workflow/tasks/ExecuteSqlScript.java | 2 +-
.../baremaps/workflow/tasks/ExportVectorTiles.java | 35 +-
.../baremaps/workflow/tasks/ImportGeoPackage.java | 2 +-
.../workflow/tasks/ImportOpenStreetMap.java | 2 +-
.../baremaps/workflow/tasks/ImportShapefile.java | 2 +-
.../workflow/tasks/UpdateOpenStreetMap.java | 2 +-
.../baremaps/tilestore/pmtiles/PMTilesReader.java | 77 ---
.../baremaps/tilestore/pmtiles/PMTilesTest.java | 125 ++---
38 files changed, 1629 insertions(+), 748 deletions(-)
delete mode 100644 baremaps-cli/licenseBundledDeps.ftl
create mode 100644 baremaps-cli/src/license/bundle.ftl
create mode 100644 baremaps-cli/src/license/override.ftl
create mode 100644 baremaps-cli/src/license/override.properties
create mode 100644 baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java
copy baremaps-core/src/main/java/org/apache/baremaps/tilestore/{file/FileTileStore.java => pmtiles/PMTilesStore.java} (59%)
create mode 100644 baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesWriter.java
create mode 100644 baremaps-core/src/main/java/org/apache/baremaps/vectortile/tileset/Database.java
delete mode 100644 baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java
[incubator-baremaps] 02/08: Implement header serialization and deserialization
Posted by bc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bchapuis pushed a commit to branch pmtiles
in repository https://gitbox.apache.org/repos/asf/incubator-baremaps.git
commit 4c91cc6f66114e294f36a7f19bdab7cd5039ba8c
Author: Bertil Chapuis <bc...@gmail.com>
AuthorDate: Tue Sep 5 23:24:54 2023 +0200
Implement header serialization and deserialization
---
.../apache/baremaps/tilestore/pmtiles/PMTiles.java | 116 +++++++++++++++++++++
.../baremaps/tilestore/pmtiles/PMTilesTest.java | 73 +++++++++++++
.../src/test/resources/pmtiles/empty.pmtiles | 0
.../src/test/resources/pmtiles/invalid.pmtiles | 1 +
.../src/test/resources/pmtiles/invalid_v4.pmtiles | Bin 0 -> 468 bytes
.../test/resources/pmtiles/test_fixture_1.pmtiles | Bin 0 -> 468 bytes
.../test/resources/pmtiles/test_fixture_2.pmtiles | Bin 0 -> 466 bytes
7 files changed, 190 insertions(+)
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
index c84b19b2..f773c7eb 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
@@ -3,6 +3,7 @@ package org.apache.baremaps.tilestore.pmtiles;
import com.google.common.math.LongMath;
import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
public class PMTiles {
@@ -142,4 +143,119 @@ public class PMTiles {
}
throw new RuntimeException("Tile zoom level exceeds max safe number limit (26)");
}
+
+ enum Compression {
+ Unknown,
+ None,
+ Gzip,
+ Brotli,
+ Zstd,
+ }
+
+ enum TileType {
+ Unknown,
+ Mvt,
+ Png,
+ Jpeg,
+ Webp,
+ Avif,
+ }
+
+ public record Header(
+ int specVersion,
+ long rootDirectoryOffset,
+ long rootDirectoryLength,
+ long jsonMetadataOffset,
+ long jsonMetadataLength,
+ long leafDirectoryOffset,
+ long leafDirectoryLength,
+ long tileDataOffset,
+ long tileDataLength,
+ long numAddressedTiles,
+ long numTileEntries,
+ long numTileContents,
+ boolean clustered,
+ Compression internalCompression,
+ Compression tileCompression,
+ TileType tileType,
+ int minZoom,
+ int maxZoom,
+ double minLon,
+ double minLat,
+ double maxLon,
+ double maxLat,
+ int centerZoom,
+ double centerLon,
+ double centerLat,
+ String etag
+ ) {}
+
+ public static Header bytesToHeader(ByteBuffer buf, String etag) {
+ buf.order(ByteOrder.LITTLE_ENDIAN);
+ return new Header(
+ buf.get(7),
+ buf.getLong(8),
+ buf.getLong(16),
+ buf.getLong(24),
+ buf.getLong(32),
+ buf.getLong(40),
+ buf.getLong(48),
+ buf.getLong(56),
+ buf.getLong(64),
+ buf.getLong(72),
+ buf.getLong(80),
+ buf.getLong(88),
+ buf.get(96) == 1,
+ Compression.values()[buf.get(97)],
+ Compression.values()[buf.get(98)],
+ TileType.values()[buf.get(99)],
+ buf.get(100),
+ buf.get(101),
+ (double) buf.getInt(102) / 10000000,
+ (double) buf.getInt(106) / 10000000,
+ (double) buf.getInt(110) / 10000000,
+ (double) buf.getInt(114) / 10000000,
+ buf.get(118),
+ (double) buf.getInt(119) / 10000000,
+ (double) buf.getInt(123) / 10000000,
+ etag
+ );
+ }
+
+ public static void headerToBytes(Header header, ByteBuffer buf) {
+ buf.order(ByteOrder.LITTLE_ENDIAN);
+ buf.put(0, (byte) 0x4d);
+ buf.put(1, (byte) 0x42);
+ buf.put(2, (byte) 0x54);
+ buf.put(3, (byte) 0x42);
+ buf.put(4, (byte) 0x49);
+ buf.put(5, (byte) 0x4e);
+ buf.put(6, (byte) 0x41);
+ buf.put(7, (byte) header.specVersion);
+ buf.putLong(8, header.rootDirectoryOffset);
+ buf.putLong(16, header.rootDirectoryLength);
+ buf.putLong(24, header.jsonMetadataOffset);
+ buf.putLong(32, header.jsonMetadataLength);
+ buf.putLong(40, header.leafDirectoryOffset);
+ buf.putLong(48, header.leafDirectoryLength);
+ buf.putLong(56, header.tileDataOffset);
+ buf.putLong(64, header.tileDataLength);
+ buf.putLong(72, header.numAddressedTiles);
+ buf.putLong(80, header.numTileEntries);
+ buf.putLong(88, header.numTileContents);
+ buf.put(96, (byte) (header.clustered ? 1 : 0));
+ buf.put(97, (byte) header.internalCompression.ordinal());
+ buf.put(98, (byte) header.tileCompression.ordinal());
+ buf.put(99, (byte) header.tileType.ordinal());
+ buf.put(100, (byte) header.minZoom);
+ buf.put(101, (byte) header.maxZoom);
+ buf.putInt(102, (int) (header.minLon * 10000000));
+ buf.putInt(106, (int) (header.minLat * 10000000));
+ buf.putInt(110, (int) (header.maxLon * 10000000));
+ buf.putInt(114, (int) (header.maxLat * 10000000));
+ buf.put(118, (byte) header.centerZoom);
+ buf.putInt(119, (int) (header.centerLon * 10000000));
+ buf.putInt(123, (int) (header.centerLat * 10000000));
+ }
+
}
diff --git a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
index c119a882..3eda409b 100644
--- a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
+++ b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
@@ -1,9 +1,14 @@
package org.apache.baremaps.tilestore.pmtiles;
import com.google.common.math.LongMath;
+import org.apache.baremaps.testing.TestFiles;
+import org.apache.baremaps.tilestore.pmtiles.PMTiles.Compression;
+import org.apache.baremaps.tilestore.pmtiles.PMTiles.TileType;
import org.junit.jupiter.api.Test;
+import java.io.IOException;
import java.nio.ByteBuffer;
+import java.nio.file.Files;
import static org.junit.jupiter.api.Assertions.*;
@@ -85,4 +90,72 @@ class PMTilesTest {
assertThrows(RuntimeException.class, () -> PMTiles.zxyToTileId(0, 1, 1));
}
+ @Test
+ void bytesToHeader() throws IOException {
+ var file = TestFiles.resolve("pmtiles/test_fixture_1.pmtiles");
+ try (var channel = Files.newByteChannel(file)) {
+ var buffer = ByteBuffer.allocate(127);
+ channel.read(buffer);
+ buffer.flip();
+ var header = PMTiles.bytesToHeader(buffer, "1");
+ assertEquals(header.rootDirectoryOffset(), 127);
+ assertEquals(header.rootDirectoryLength(), 25);
+ assertEquals(header.jsonMetadataOffset(), 152);
+ assertEquals(header.jsonMetadataLength(), 247);
+ assertEquals(header.leafDirectoryOffset(), 0);
+ assertEquals(header.leafDirectoryLength(), 0);
+ assertEquals(header.tileDataOffset(), 399);
+ assertEquals(header.tileDataLength(), 69);
+ assertEquals(header.numAddressedTiles(), 1);
+ assertEquals(header.numTileEntries(), 1);
+ assertEquals(header.numTileContents(), 1);
+ assertFalse(header.clustered());
+ assertEquals(header.internalCompression(), Compression.Gzip);
+ assertEquals(header.tileCompression(), Compression.Gzip);
+ assertEquals(header.tileType(), TileType.Mvt);
+ assertEquals(header.minZoom(), 0);
+ assertEquals(header.maxZoom(), 0);
+ assertEquals(header.minLon(), 0);
+ assertEquals(header.minLat(), 0);
+ assertEquals(Math.round(header.maxLon()), 1);
+ assertEquals(Math.round(header.maxLat()), 1);
+ }
+ }
+
+ @Test
+ void headerToBytes() throws IOException {
+ var etag = "1";
+ var buffer = ByteBuffer.allocate(127);
+ var header = new PMTiles.Header(
+ 127,
+ 25,
+ 152,
+ 247,
+ 0,
+ 0,
+ 399,
+ 69,
+ 1,
+ 1,
+ 1,
+ 10,
+ false,
+ Compression.Gzip,
+ Compression.Gzip,
+ TileType.Mvt,
+ 0,
+ 0,
+ 0,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ etag);
+ PMTiles.headerToBytes(header, buffer);
+ var header2 = PMTiles.bytesToHeader(buffer, etag);
+ assertEquals(header, header2);
+ }
+
}
\ No newline at end of file
diff --git a/baremaps-core/src/test/resources/pmtiles/empty.pmtiles b/baremaps-core/src/test/resources/pmtiles/empty.pmtiles
new file mode 100644
index 00000000..e69de29b
diff --git a/baremaps-core/src/test/resources/pmtiles/invalid.pmtiles b/baremaps-core/src/test/resources/pmtiles/invalid.pmtiles
new file mode 100644
index 00000000..f2b720ba
--- /dev/null
+++ b/baremaps-core/src/test/resources/pmtiles/invalid.pmtiles
@@ -0,0 +1 @@
+This is an invalid tile archive, a test case to make sure that the code throws an error, but it needs to be the minimum size to pass the first test
diff --git a/baremaps-core/src/test/resources/pmtiles/invalid_v4.pmtiles b/baremaps-core/src/test/resources/pmtiles/invalid_v4.pmtiles
new file mode 100644
index 00000000..1871cb27
Binary files /dev/null and b/baremaps-core/src/test/resources/pmtiles/invalid_v4.pmtiles differ
diff --git a/baremaps-core/src/test/resources/pmtiles/test_fixture_1.pmtiles b/baremaps-core/src/test/resources/pmtiles/test_fixture_1.pmtiles
new file mode 100644
index 00000000..c86db1f2
Binary files /dev/null and b/baremaps-core/src/test/resources/pmtiles/test_fixture_1.pmtiles differ
diff --git a/baremaps-core/src/test/resources/pmtiles/test_fixture_2.pmtiles b/baremaps-core/src/test/resources/pmtiles/test_fixture_2.pmtiles
new file mode 100644
index 00000000..cb19dd5f
Binary files /dev/null and b/baremaps-core/src/test/resources/pmtiles/test_fixture_2.pmtiles differ
[incubator-baremaps] 05/08: Fix unit tests
Posted by bc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bchapuis pushed a commit to branch pmtiles
in repository https://gitbox.apache.org/repos/asf/incubator-baremaps.git
commit dbae519e893dd7a3cf32d3156f764c7f5877ce97
Author: Bertil Chapuis <bc...@gmail.com>
AuthorDate: Thu Sep 7 11:03:51 2023 +0200
Fix unit tests
---
.../java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
index b5bea852..12adeaac 100644
--- a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
+++ b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
@@ -19,7 +19,6 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.util.ArrayList;
-import java.util.Optional;
import org.apache.baremaps.testing.TestFiles;
import org.apache.baremaps.tilestore.pmtiles.PMTiles.Compression;
import org.apache.baremaps.tilestore.pmtiles.PMTiles.Entry;
@@ -192,7 +191,7 @@ class PMTilesTest {
@Test
void searchForMissingEntry() {
var entries = new ArrayList<Entry>();
- assertEquals(PMTiles.findTile(entries, 101), Optional.empty());
+ assertEquals(PMTiles.findTile(entries, 101), null);
}
@Test
@@ -200,7 +199,7 @@ class PMTilesTest {
var entry = new Entry(100, 1, 1, 1);
var entries = new ArrayList<Entry>();
entries.add(entry);
- assertEquals(PMTiles.findTile(entries, 100), Optional.of(entry));
+ assertEquals(PMTiles.findTile(entries, 100), entry);
}
@Test
@@ -209,7 +208,7 @@ class PMTilesTest {
var entries = new ArrayList<Entry>();
entries.add(entry);
entries.add(new Entry(5, 5, 1, 2));
- assertEquals(PMTiles.findTile(entries, 4), Optional.of(entry));
+ assertEquals(PMTiles.findTile(entries, 4), entry);
}
@Test
[incubator-baremaps] 06/08: Refactor package and introduce a PMTilesReader class
Posted by bc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bchapuis pushed a commit to branch pmtiles
in repository https://gitbox.apache.org/repos/asf/incubator-baremaps.git
commit 39f811d5233d7a4b37fdf31f96e8e1f53755d18c
Author: Bertil Chapuis <bc...@gmail.com>
AuthorDate: Thu Sep 7 17:25:58 2023 +0200
Refactor package and introduce a PMTilesReader class
---
.../baremaps/tilestore/pmtiles/Compression.java | 70 ++++++
.../apache/baremaps/tilestore/pmtiles/Entry.java | 51 ++++
.../apache/baremaps/tilestore/pmtiles/Header.java | 258 +++++++++++++++++++++
.../apache/baremaps/tilestore/pmtiles/PMTiles.java | 160 +++----------
.../baremaps/tilestore/pmtiles/TileType.java | 10 +
.../baremaps/tilestore/pmtiles/PMTilesReader.java | 77 ++++++
.../baremaps/tilestore/pmtiles/PMTilesTest.java | 84 ++++---
.../baremaps/vectortile/VectorTileViewer.java | 2 +-
8 files changed, 554 insertions(+), 158 deletions(-)
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Compression.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Compression.java
new file mode 100644
index 00000000..4360d9c7
--- /dev/null
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Compression.java
@@ -0,0 +1,70 @@
+package org.apache.baremaps.tilestore.pmtiles;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+enum Compression {
+ Unknown,
+ None,
+ Gzip,
+ Brotli,
+ Zstd;
+
+ ByteBuffer decompress(ByteBuffer buffer) {
+ return switch (this) {
+ case None -> buffer;
+ case Gzip -> decompressGzip(buffer);
+ case Brotli -> decompressBrotli(buffer);
+ case Zstd -> decompressZstd(buffer);
+ default -> throw new RuntimeException("Unknown compression");
+ };
+ }
+
+ static ByteBuffer decompressGzip(ByteBuffer buffer) {
+ try(var inputStream = new GZIPInputStream(new ByteArrayInputStream(buffer.array()))) {
+ return ByteBuffer.wrap(inputStream.readAllBytes());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ static ByteBuffer decompressBrotli(ByteBuffer buffer) {
+ throw new RuntimeException("Brotli compression not implemented");
+ }
+
+ static ByteBuffer decompressZstd(ByteBuffer buffer) {
+ throw new RuntimeException("Zstd compression not implemented");
+ }
+
+ ByteBuffer compress(ByteBuffer buffer) {
+ return switch (this) {
+ case None -> buffer;
+ case Gzip -> compressGzip(buffer);
+ case Brotli -> compressBrotli(buffer);
+ case Zstd -> compressZstd(buffer);
+ default -> throw new RuntimeException("Unknown compression");
+ };
+ }
+
+ static ByteBuffer compressGzip(ByteBuffer buffer) {
+ try(var outputStream = new ByteArrayOutputStream();
+ var gzipOutputStream = new GZIPOutputStream(outputStream)) {
+ gzipOutputStream.write(buffer.array());
+ return ByteBuffer.wrap(outputStream.toByteArray());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ static ByteBuffer compressBrotli(ByteBuffer buffer) {
+ throw new RuntimeException("Brotli compression not implemented");
+ }
+
+ static ByteBuffer compressZstd(ByteBuffer buffer) {
+ throw new RuntimeException("Zstd compression not implemented");
+ }
+}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Entry.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Entry.java
new file mode 100644
index 00000000..97fe00e2
--- /dev/null
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Entry.java
@@ -0,0 +1,51 @@
+package org.apache.baremaps.tilestore.pmtiles;
+
+public class Entry {
+ private long tileId;
+ private long offset;
+ private long length;
+ private long runLength;
+
+ public Entry() {
+
+ }
+
+ public Entry(long tileId, long offset, long length, long runLength) {
+ this.tileId = tileId;
+ this.offset = offset;
+ this.length = length;
+ this.runLength = runLength;
+ }
+
+ public long getTileId() {
+ return tileId;
+ }
+
+ public void setTileId(long tileId) {
+ this.tileId = tileId;
+ }
+
+ public long getOffset() {
+ return offset;
+ }
+
+ public void setOffset(long offset) {
+ this.offset = offset;
+ }
+
+ public long getLength() {
+ return length;
+ }
+
+ public void setLength(long length) {
+ this.length = length;
+ }
+
+ public long getRunLength() {
+ return runLength;
+ }
+
+ public void setRunLength(long runLength) {
+ this.runLength = runLength;
+ }
+}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Header.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Header.java
new file mode 100644
index 00000000..d3cb42cb
--- /dev/null
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Header.java
@@ -0,0 +1,258 @@
+package org.apache.baremaps.tilestore.pmtiles;
+
+public class Header {
+
+ private int specVersion;
+ private long rootDirectoryOffset;
+ private long rootDirectoryLength;
+ private long jsonMetadataOffset;
+ private long jsonMetadataLength;
+ private long leafDirectoryOffset;
+ private long leafDirectoryLength;
+ private long tileDataOffset;
+ private long tileDataLength;
+ private long numAddressedTiles;
+ private long numTileEntries;
+ private long numTileContents;
+ private boolean clustered;
+ private Compression internalCompression;
+ private Compression tileCompression;
+ private TileType tileType;
+ private int minZoom;
+ private int maxZoom;
+ private double minLon;
+ private double minLat;
+ private double maxLon;
+ private double maxLat;
+ private int centerZoom;
+ private double centerLon;
+ private double centerLat;
+
+ public Header(int specVersion, long rootDirectoryOffset, long rootDirectoryLength, long jsonMetadataOffset, long jsonMetadataLength, long leafDirectoryOffset, long leafDirectoryLength, long tileDataOffset, long tileDataLength, long numAddressedTiles, long numTileEntries, long numTileContents, boolean clustered, Compression internalCompression, Compression tileCompression, TileType tileType, int minZoom, int maxZoom, double minLon, double minLat, double maxLon, double maxLat, int cent [...]
+ this.specVersion = specVersion;
+ this.rootDirectoryOffset = rootDirectoryOffset;
+ this.rootDirectoryLength = rootDirectoryLength;
+ this.jsonMetadataOffset = jsonMetadataOffset;
+ this.jsonMetadataLength = jsonMetadataLength;
+ this.leafDirectoryOffset = leafDirectoryOffset;
+ this.leafDirectoryLength = leafDirectoryLength;
+ this.tileDataOffset = tileDataOffset;
+ this.tileDataLength = tileDataLength;
+ this.numAddressedTiles = numAddressedTiles;
+ this.numTileEntries = numTileEntries;
+ this.numTileContents = numTileContents;
+ this.clustered = clustered;
+ this.internalCompression = internalCompression;
+ this.tileCompression = tileCompression;
+ this.tileType = tileType;
+ this.minZoom = minZoom;
+ this.maxZoom = maxZoom;
+ this.minLon = minLon;
+ this.minLat = minLat;
+ this.maxLon = maxLon;
+ this.maxLat = maxLat;
+ this.centerZoom = centerZoom;
+ this.centerLon = centerLon;
+ this.centerLat = centerLat;
+ }
+
+ public int getSpecVersion() {
+ return specVersion;
+ }
+
+ public void setSpecVersion(int specVersion) {
+ this.specVersion = specVersion;
+ }
+
+ public long getRootDirectoryOffset() {
+ return rootDirectoryOffset;
+ }
+
+ public void setRootDirectoryOffset(long rootDirectoryOffset) {
+ this.rootDirectoryOffset = rootDirectoryOffset;
+ }
+
+ public long getRootDirectoryLength() {
+ return rootDirectoryLength;
+ }
+
+ public void setRootDirectoryLength(long rootDirectoryLength) {
+ this.rootDirectoryLength = rootDirectoryLength;
+ }
+
+ public long getJsonMetadataOffset() {
+ return jsonMetadataOffset;
+ }
+
+ public void setJsonMetadataOffset(long jsonMetadataOffset) {
+ this.jsonMetadataOffset = jsonMetadataOffset;
+ }
+
+ public long getJsonMetadataLength() {
+ return jsonMetadataLength;
+ }
+
+ public void setJsonMetadataLength(long jsonMetadataLength) {
+ this.jsonMetadataLength = jsonMetadataLength;
+ }
+
+ public long getLeafDirectoryOffset() {
+ return leafDirectoryOffset;
+ }
+
+ public void setLeafDirectoryOffset(long leafDirectoryOffset) {
+ this.leafDirectoryOffset = leafDirectoryOffset;
+ }
+
+ public long getLeafDirectoryLength() {
+ return leafDirectoryLength;
+ }
+
+ public void setLeafDirectoryLength(long leafDirectoryLength) {
+ this.leafDirectoryLength = leafDirectoryLength;
+ }
+
+ public long getTileDataOffset() {
+ return tileDataOffset;
+ }
+
+ public void setTileDataOffset(long tileDataOffset) {
+ this.tileDataOffset = tileDataOffset;
+ }
+
+ public long getTileDataLength() {
+ return tileDataLength;
+ }
+
+ public void setTileDataLength(long tileDataLength) {
+ this.tileDataLength = tileDataLength;
+ }
+
+ public long getNumAddressedTiles() {
+ return numAddressedTiles;
+ }
+
+ public void setNumAddressedTiles(long numAddressedTiles) {
+ this.numAddressedTiles = numAddressedTiles;
+ }
+
+ public long getNumTileEntries() {
+ return numTileEntries;
+ }
+
+ public void setNumTileEntries(long numTileEntries) {
+ this.numTileEntries = numTileEntries;
+ }
+
+ public long getNumTileContents() {
+ return numTileContents;
+ }
+
+ public void setNumTileContents(long numTileContents) {
+ this.numTileContents = numTileContents;
+ }
+
+ public boolean isClustered() {
+ return clustered;
+ }
+
+ public void setClustered(boolean clustered) {
+ this.clustered = clustered;
+ }
+
+ public Compression getInternalCompression() {
+ return internalCompression;
+ }
+
+ public void setInternalCompression(Compression internalCompression) {
+ this.internalCompression = internalCompression;
+ }
+
+ public Compression getTileCompression() {
+ return tileCompression;
+ }
+
+ public void setTileCompression(Compression tileCompression) {
+ this.tileCompression = tileCompression;
+ }
+
+ public TileType getTileType() {
+ return tileType;
+ }
+
+ public void setTileType(TileType tileType) {
+ this.tileType = tileType;
+ }
+
+ public int getMinZoom() {
+ return minZoom;
+ }
+
+ public void setMinZoom(int minZoom) {
+ this.minZoom = minZoom;
+ }
+
+ public int getMaxZoom() {
+ return maxZoom;
+ }
+
+ public void setMaxZoom(int maxZoom) {
+ this.maxZoom = maxZoom;
+ }
+
+ public double getMinLon() {
+ return minLon;
+ }
+
+ public void setMinLon(double minLon) {
+ this.minLon = minLon;
+ }
+
+ public double getMinLat() {
+ return minLat;
+ }
+
+ public void setMinLat(double minLat) {
+ this.minLat = minLat;
+ }
+
+ public double getMaxLon() {
+ return maxLon;
+ }
+
+ public void setMaxLon(double maxLon) {
+ this.maxLon = maxLon;
+ }
+
+ public double getMaxLat() {
+ return maxLat;
+ }
+
+ public void setMaxLat(double maxLat) {
+ this.maxLat = maxLat;
+ }
+
+ public int getCenterZoom() {
+ return centerZoom;
+ }
+
+ public void setCenterZoom(int centerZoom) {
+ this.centerZoom = centerZoom;
+ }
+
+ public double getCenterLon() {
+ return centerLon;
+ }
+
+ public void setCenterLon(double centerLon) {
+ this.centerLon = centerLon;
+ }
+
+ public double getCenterLat() {
+ return centerLat;
+ }
+
+ public void setCenterLat(double centerLat) {
+ this.centerLat = centerLat;
+ }
+}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
index 1ff45353..7d260a5b 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
@@ -168,55 +168,9 @@ public class PMTiles {
throw new RuntimeException("Tile zoom level exceeds max safe number limit (26)");
}
- enum Compression {
- Unknown,
- None,
- Gzip,
- Brotli,
- Zstd,
- }
-
- enum TileType {
- Unknown,
- Mvt,
- Png,
- Jpeg,
- Webp,
- Avif,
- }
-
private static final int HEADER_SIZE_BYTES = 127;
- public record Header(
- int specVersion,
- long rootDirectoryOffset,
- long rootDirectoryLength,
- long jsonMetadataOffset,
- long jsonMetadataLength,
- long leafDirectoryOffset,
- long leafDirectoryLength,
- long tileDataOffset,
- long tileDataLength,
- long numAddressedTiles,
- long numTileEntries,
- long numTileContents,
- boolean clustered,
- Compression internalCompression,
- Compression tileCompression,
- TileType tileType,
- int minZoom,
- int maxZoom,
- double minLon,
- double minLat,
- double maxLon,
- double maxLat,
- int centerZoom,
- double centerLon,
- double centerLat,
- String etag) {
- }
-
- public static Header decodeHeader(ByteBuffer buf, String etag) {
+ public static Header decodeHeader(ByteBuffer buf) {
buf.order(ByteOrder.LITTLE_ENDIAN);
return new Header(
buf.get(7),
@@ -243,8 +197,8 @@ public class PMTiles {
(double) buf.getInt(114) / 10000000,
buf.get(118),
(double) buf.getInt(119) / 10000000,
- (double) buf.getInt(123) / 10000000,
- etag);
+ (double) buf.getInt(123) / 10000000
+ );
}
public static void encodeHeader(Header header, ByteBuffer buf) {
@@ -256,81 +210,31 @@ public class PMTiles {
buf.put(4, (byte) 0x6C);
buf.put(5, (byte) 0x65);
buf.put(6, (byte) 0x73);
- buf.put(7, (byte) header.specVersion);
- buf.putLong(8, header.rootDirectoryOffset);
- buf.putLong(16, header.rootDirectoryLength);
- buf.putLong(24, header.jsonMetadataOffset);
- buf.putLong(32, header.jsonMetadataLength);
- buf.putLong(40, header.leafDirectoryOffset);
- buf.putLong(48, header.leafDirectoryLength);
- buf.putLong(56, header.tileDataOffset);
- buf.putLong(64, header.tileDataLength);
- buf.putLong(72, header.numAddressedTiles);
- buf.putLong(80, header.numTileEntries);
- buf.putLong(88, header.numTileContents);
- buf.put(96, (byte) (header.clustered ? 1 : 0));
- buf.put(97, (byte) header.internalCompression.ordinal());
- buf.put(98, (byte) header.tileCompression.ordinal());
- buf.put(99, (byte) header.tileType.ordinal());
- buf.put(100, (byte) header.minZoom);
- buf.put(101, (byte) header.maxZoom);
- buf.putInt(102, (int) (header.minLon * 10000000));
- buf.putInt(106, (int) (header.minLat * 10000000));
- buf.putInt(110, (int) (header.maxLon * 10000000));
- buf.putInt(114, (int) (header.maxLat * 10000000));
- buf.put(118, (byte) header.centerZoom);
- buf.putInt(119, (int) (header.centerLon * 10000000));
- buf.putInt(123, (int) (header.centerLat * 10000000));
- }
-
- public static class Entry {
- private long tileId;
- private long offset;
- private long length;
- private long runLength;
-
- public Entry() {
-
- }
-
- public Entry(long tileId, long offset, long length, long runLength) {
- this.tileId = tileId;
- this.offset = offset;
- this.length = length;
- this.runLength = runLength;
- }
-
- public long getTileId() {
- return tileId;
- }
-
- public void setTileId(long tileId) {
- this.tileId = tileId;
- }
-
- public long getOffset() {
- return offset;
- }
-
- public void setOffset(long offset) {
- this.offset = offset;
- }
-
- public long getLength() {
- return length;
- }
-
- public void setLength(long length) {
- this.length = length;
- }
-
- public long getRunLength() {
- return runLength;
- }
-
- public void setRunLength(long runLength) {
- this.runLength = runLength;
- }
+ buf.put(7, (byte) header.getSpecVersion());
+ buf.putLong(8, header.getRootDirectoryOffset());
+ buf.putLong(16, header.getRootDirectoryLength());
+ buf.putLong(24, header.getJsonMetadataOffset());
+ buf.putLong(32, header.getJsonMetadataLength());
+ buf.putLong(40, header.getLeafDirectoryOffset());
+ buf.putLong(48, header.getLeafDirectoryLength());
+ buf.putLong(56, header.getTileDataOffset());
+ buf.putLong(64, header.getTileDataLength());
+ buf.putLong(72, header.getNumAddressedTiles());
+ buf.putLong(80, header.getNumTileEntries());
+ buf.putLong(88, header.getNumTileContents());
+ buf.put(96, (byte) (header.isClustered() ? 1 : 0));
+ buf.put(97, (byte) header.getInternalCompression().ordinal());
+ buf.put(98, (byte) header.getTileCompression().ordinal());
+ buf.put(99, (byte) header.getTileType().ordinal());
+ buf.put(100, (byte) header.getMinZoom());
+ buf.put(101, (byte) header.getMaxZoom());
+ buf.putInt(102, (int) (header.getMinLon() * 10000000));
+ buf.putInt(106, (int) (header.getMinLat() * 10000000));
+ buf.putInt(110, (int) (header.getMaxLon() * 10000000));
+ buf.putInt(114, (int) (header.getMaxLat() * 10000000));
+ buf.put(118, (byte) header.getCenterZoom());
+ buf.putInt(119, (int) (header.getCenterLon() * 10000000));
+ buf.putInt(123, (int) (header.getCenterLat() * 10000000));
}
public static void encodeDirectory(ByteBuffer buffer, List<Entry> entries) {
@@ -350,7 +254,7 @@ public class PMTiles {
for (Entry entry : entries) {
if (entry.getOffset() == 0 && entry.getLength() > 0) {
Entry prevEntry = entries.get(entries.indexOf(entry) - 1);
- encodeVarInt(buffer, prevEntry.offset + prevEntry.length + 1);
+ encodeVarInt(buffer, prevEntry.getOffset() + prevEntry.getLength() + 1);
} else {
encodeVarInt(buffer, entry.getOffset() + 1);
}
@@ -381,7 +285,7 @@ public class PMTiles {
long value = decodeVarInt(buffer);
if (value == 0 && i > 0) {
Entry prevEntry = entries.get(i - 1);
- entries.get(i).setOffset(prevEntry.offset + prevEntry.length);;
+ entries.get(i).setOffset(prevEntry.getOffset() + prevEntry.getLength());;
} else {
entries.get(i).setOffset(value - 1);
}
@@ -406,10 +310,10 @@ public class PMTiles {
// at this point, m > n
if (n >= 0) {
- if (entries.get(n).runLength == 0) {
+ if (entries.get(n).getRunLength() == 0) {
return entries.get(n);
}
- if (tileId - entries.get(n).tileId < entries.get(n).runLength) {
+ if (tileId - entries.get(n).getTileId() < entries.get(n).getRunLength()) {
return entries.get(n);
}
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/TileType.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/TileType.java
new file mode 100644
index 00000000..10bf41c8
--- /dev/null
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/TileType.java
@@ -0,0 +1,10 @@
+package org.apache.baremaps.tilestore.pmtiles;
+
+enum TileType {
+ Unknown,
+ Mvt,
+ Png,
+ Jpeg,
+ Webp,
+ Avif,
+}
diff --git a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java
new file mode 100644
index 00000000..4ca92112
--- /dev/null
+++ b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java
@@ -0,0 +1,77 @@
+package org.apache.baremaps.tilestore.pmtiles;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+
+public class PMTilesReader {
+
+ private final Path path;
+
+ private Header header;
+
+ private List<Entry> rootEntries;
+
+ public PMTilesReader(Path path) {
+ this.path = path;
+ }
+
+ public Header getHeader() {
+ if (header == null) {
+ try (var channel = Files.newByteChannel(path)) {
+ var buffer = ByteBuffer.allocate(127);
+ channel.read(buffer);
+ buffer.flip();
+ header = PMTiles.decodeHeader(buffer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ return header;
+ }
+
+ public List<Entry> getRootDirectory() {
+ if (rootEntries == null) {
+ var header = getHeader();
+ rootEntries = getDirectory(header.getRootDirectoryOffset(), (int) header.getRootDirectoryLength());
+ }
+ return rootEntries;
+ }
+
+ public List<Entry> getDirectory(long offset, int length) {
+ var header = getHeader();
+ try (var channel = Files.newByteChannel(path)) {
+ var compressed = ByteBuffer.allocate(length);
+ channel.position(offset);
+ channel.read(compressed);
+ compressed.flip();
+ var decompressed = header.getInternalCompression().decompress(compressed);
+ return PMTiles.decodeDirectory(decompressed);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public ByteBuffer getTile(int z, long x, long y) {
+ var tileId = PMTiles.zxyToTileId(z, x, y);
+ var header = getHeader();
+ var entries = getRootDirectory();
+ var entry = PMTiles.findTile(entries, tileId);
+
+ if (entry == null) {
+ return null;
+ }
+
+ try (var channel = Files.newByteChannel(path)) {
+ var compressed = ByteBuffer.allocate((int) entry.getLength());
+ channel.position(header.getTileDataOffset() + entry.getOffset());
+ channel.read(compressed);
+ compressed.flip();
+ return header.getTileCompression().decompress(compressed);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+}
diff --git a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
index 12adeaac..84211805 100644
--- a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
+++ b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
@@ -19,12 +19,16 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.util.ArrayList;
+
+import com.google.protobuf.InvalidProtocolBufferException;
import org.apache.baremaps.testing.TestFiles;
-import org.apache.baremaps.tilestore.pmtiles.PMTiles.Compression;
-import org.apache.baremaps.tilestore.pmtiles.PMTiles.Entry;
-import org.apache.baremaps.tilestore.pmtiles.PMTiles.TileType;
+import org.apache.baremaps.vectortile.VectorTileDecoder;
+import org.apache.baremaps.vectortile.VectorTileViewer;
+import org.apache.baremaps.vectortile.VectorTileViewer.TilePanel;
import org.junit.jupiter.api.Test;
+import javax.swing.*;
+
class PMTilesTest {
@Test
@@ -127,28 +131,28 @@ class PMTilesTest {
var buffer = ByteBuffer.allocate(127);
channel.read(buffer);
buffer.flip();
- var header = PMTiles.decodeHeader(buffer, "1");
- assertEquals(header.rootDirectoryOffset(), 127);
- assertEquals(header.rootDirectoryLength(), 25);
- assertEquals(header.jsonMetadataOffset(), 152);
- assertEquals(header.jsonMetadataLength(), 247);
- assertEquals(header.leafDirectoryOffset(), 0);
- assertEquals(header.leafDirectoryLength(), 0);
- assertEquals(header.tileDataOffset(), 399);
- assertEquals(header.tileDataLength(), 69);
- assertEquals(header.numAddressedTiles(), 1);
- assertEquals(header.numTileEntries(), 1);
- assertEquals(header.numTileContents(), 1);
- assertFalse(header.clustered());
- assertEquals(header.internalCompression(), Compression.Gzip);
- assertEquals(header.tileCompression(), Compression.Gzip);
- assertEquals(header.tileType(), TileType.Mvt);
- assertEquals(header.minZoom(), 0);
- assertEquals(header.maxZoom(), 0);
- assertEquals(header.minLon(), 0);
- assertEquals(header.minLat(), 0);
- assertEquals(Math.round(header.maxLon()), 1);
- assertEquals(Math.round(header.maxLat()), 1);
+ var header = PMTiles.decodeHeader(buffer);
+ assertEquals(header.getRootDirectoryOffset(), 127);
+ assertEquals(header.getRootDirectoryLength(), 25);
+ assertEquals(header.getJsonMetadataOffset(), 152);
+ assertEquals(header.getJsonMetadataLength(), 247);
+ assertEquals(header.getLeafDirectoryOffset(), 0);
+ assertEquals(header.getLeafDirectoryLength(), 0);
+ assertEquals(header.getTileDataOffset(), 399);
+ assertEquals(header.getTileDataLength(), 69);
+ assertEquals(header.getNumAddressedTiles(), 1);
+ assertEquals(header.getNumTileEntries(), 1);
+ assertEquals(header.getNumTileContents(), 1);
+ assertFalse(header.isClustered());
+ assertEquals(header.getInternalCompression(), Compression.Gzip);
+ assertEquals(header.getTileCompression(), Compression.Gzip);
+ assertEquals(header.getTileType(), TileType.Mvt);
+ assertEquals(header.getMinZoom(), 0);
+ assertEquals(header.getMaxZoom(), 0);
+ assertEquals(header.getMinLon(), 0);
+ assertEquals(header.getMinLat(), 0);
+ assertEquals(Math.round(header.getMaxLon()), 1);
+ assertEquals(Math.round(header.getMaxLat()), 1);
}
}
@@ -156,7 +160,7 @@ class PMTilesTest {
void encodeHeader() throws IOException {
var etag = "1";
var buffer = ByteBuffer.allocate(127);
- var header = new PMTiles.Header(
+ var header = new Header(
127,
25,
152,
@@ -181,10 +185,10 @@ class PMTilesTest {
0,
0,
0,
- 0,
- etag);
+ 0
+ );
PMTiles.encodeHeader(header, buffer);
- var header2 = PMTiles.decodeHeader(buffer, etag);
+ var header2 = PMTiles.decodeHeader(buffer);
assertEquals(header, header2);
}
@@ -244,5 +248,27 @@ class PMTilesTest {
assertEquals(entry.getLength(), 1);
}
+ @Test
+ void reader() throws InvalidProtocolBufferException, InterruptedException {
+ var reader = new PMTilesReader(TestFiles.resolve("pmtiles/test_fixture_1.pmtiles"));
+ var header = reader.getHeader();
+ assertEquals(header.getRootDirectoryOffset(), 127);
+ var rootDirectory = reader.getRootDirectory();
+ assertEquals(rootDirectory.size(), 1);
+ var entry = rootDirectory.get(0);
+ ByteBuffer buffer = reader.getTile(0,0,0);
+
+ VectorTileViewer viewer = new VectorTileViewer();
+ var parsed = org.apache.baremaps.mvt.binary.VectorTile.Tile.parseFrom(buffer);
+ var tile = new VectorTileDecoder().decodeTile(parsed);
+ JFrame f = new JFrame("Vector Tile Viewer");
+ f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
+ f.add(new TilePanel(tile, 1000));
+ f.pack();
+ f.setVisible(true);
+
+ Thread.sleep(10000);
+ }
+
}
diff --git a/baremaps-core/src/test/java/org/apache/baremaps/vectortile/VectorTileViewer.java b/baremaps-core/src/test/java/org/apache/baremaps/vectortile/VectorTileViewer.java
index 079d70a1..244f4733 100644
--- a/baremaps-core/src/test/java/org/apache/baremaps/vectortile/VectorTileViewer.java
+++ b/baremaps-core/src/test/java/org/apache/baremaps/vectortile/VectorTileViewer.java
@@ -44,7 +44,7 @@ public class VectorTileViewer {
}
}
- static class TilePanel extends JPanel {
+ public static class TilePanel extends JPanel {
private final Tile tile;
[incubator-baremaps] 01/08: Add pmtiles functions
Posted by bc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bchapuis pushed a commit to branch pmtiles
in repository https://gitbox.apache.org/repos/asf/incubator-baremaps.git
commit a30241e2482c7d7fb4970eef7011b8328a0d1f9e
Author: Bertil Chapuis <bc...@gmail.com>
AuthorDate: Tue Sep 5 17:27:09 2023 +0200
Add pmtiles functions
---
.../apache/baremaps/tilestore/pmtiles/PMTiles.java | 145 +++++++++++++++++++++
.../baremaps/tilestore/pmtiles/PMTilesTest.java | 88 +++++++++++++
2 files changed, 233 insertions(+)
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
new file mode 100644
index 00000000..c84b19b2
--- /dev/null
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
@@ -0,0 +1,145 @@
+package org.apache.baremaps.tilestore.pmtiles;
+
+import com.google.common.math.LongMath;
+
+import java.nio.ByteBuffer;
+
+public class PMTiles {
+
+ public static long toNum(long low, long high) {
+ return high * 0x100000000L + low;
+ }
+
+ public static long readVarintRemainder(long l, ByteBuffer buf) {
+ long h, b;
+ b = buf.get() & 0xff;
+ h = (b & 0x70) >> 4;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ b = buf.get() & 0xff;
+ h |= (b & 0x7f) << 3;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ b = buf.get() & 0xff;
+ h |= (b & 0x7f) << 10;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ b = buf.get() & 0xff;
+ h |= (b & 0x7f) << 17;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ b = buf.get() & 0xff;
+ h |= (b & 0x7f) << 24;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ b = buf.get() & 0xff;
+ h |= (b & 0x01) << 31;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ throw new RuntimeException("Expected varint not more than 10 bytes");
+ }
+
+ public static long readVarint(ByteBuffer buf) {
+ long val, b;
+ b = buf.get() & 0xff;
+ val = b & 0x7f;
+ if (b < 0x80) {
+ return val;
+ }
+ b = buf.get() & 0xff;
+ val |= (b & 0x7f) << 7;
+ if (b < 0x80) {
+ return val;
+ }
+ b = buf.get() & 0xff;
+ val |= (b & 0x7f) << 14;
+ if (b < 0x80) {
+ return val;
+ }
+ b = buf.get() & 0xff;
+ val |= (b & 0x7f) << 21;
+ if (b < 0x80) {
+ return val;
+ }
+ val |= (b & 0x0f) << 28;
+ return readVarintRemainder(val, buf);
+ }
+
+ public static void rotate(long n, long[] xy, long rx, long ry) {
+ if (ry == 0) {
+ if (rx == 1) {
+ xy[0] = n - 1 - xy[0];
+ xy[1] = n - 1 - xy[1];
+ }
+ long t = xy[0];
+ xy[0] = xy[1];
+ xy[1] = t;
+ }
+ }
+
+ public static long[] idOnLevel(int z, long pos) {
+ long n = LongMath.pow(2, z);
+ long rx, ry, t = pos;
+ long[] xy = new long[]{0, 0};
+ long s = 1;
+ while (s < n) {
+ rx = 1 & (t / 2);
+ ry = 1 & (t ^ rx);
+ rotate(s, xy, rx, ry);
+ xy[0] += s * rx;
+ xy[1] += s * ry;
+ t = t / 4;
+ s *= 2;
+ }
+ return new long[]{z, xy[0], xy[1]};
+ }
+
+ private static long[] tzValues = new long[]{
+ 0, 1, 5, 21, 85, 341, 1365, 5461, 21845, 87381, 349525, 1398101, 5592405,
+ 22369621, 89478485, 357913941, 1431655765, 5726623061L, 22906492245L,
+ 91625968981L, 366503875925L, 1466015503701L, 5864062014805L, 23456248059221L,
+ 93824992236885L, 375299968947541L, 1501199875790165L,
+ };
+
+ public static long zxyToTileId(int z, long x, long y) {
+ if (z > 26) {
+ throw new RuntimeException("Tile zoom level exceeds max safe number limit (26)");
+ }
+ if (x > Math.pow(2, z) - 1 || y > Math.pow(2, z) - 1) {
+ throw new RuntimeException("tile x/y outside zoom level bounds");
+ }
+ long acc = tzValues[z];
+ long n = LongMath.pow(2, z);
+ long rx = 0;
+ long ry = 0;
+ long d = 0;
+ long[] xy = new long[]{x, y};
+ long s = n / 2;
+ while (s > 0) {
+ rx = (xy[0] & s) > 0 ? 1 : 0;
+ ry = (xy[1] & s) > 0 ? 1 : 0;
+ d += s * s * ((3 * rx) ^ ry);
+ rotate(s, xy, rx, ry);
+ s = s / 2;
+ }
+ return acc + d;
+ }
+
+ public static long[] tileIdToZxy(long i) {
+ long acc = 0;
+ for (int z = 0; z < 27; z++) {
+ long numTiles = (0x1L << z) * (0x1L << z);
+ if (acc + numTiles > i) {
+ return idOnLevel(z, i - acc);
+ }
+ acc += numTiles;
+ }
+ throw new RuntimeException("Tile zoom level exceeds max safe number limit (26)");
+ }
+}
diff --git a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
new file mode 100644
index 00000000..c119a882
--- /dev/null
+++ b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
@@ -0,0 +1,88 @@
+package org.apache.baremaps.tilestore.pmtiles;
+
+import com.google.common.math.LongMath;
+import org.junit.jupiter.api.Test;
+
+import java.nio.ByteBuffer;
+
+import static org.junit.jupiter.api.Assertions.*;
+
+class PMTilesTest {
+
+ @Test
+ void varint() {
+ var b = ByteBuffer.wrap(new byte[]{
+ (byte) 0, (byte) 1,
+ (byte) 127, (byte) 0xe5,
+ (byte) 0x8e, (byte) 0x26
+ });
+ assertEquals(PMTiles.readVarint(b), 0);
+ assertEquals(PMTiles.readVarint(b), 1);
+ assertEquals(PMTiles.readVarint(b), 127);
+ assertEquals(PMTiles.readVarint(b), 624485);
+ b = ByteBuffer.wrap(new byte[]{
+ (byte) 0xff, (byte) 0xff,
+ (byte) 0xff, (byte) 0xff,
+ (byte) 0xff, (byte) 0xff,
+ (byte) 0xff, (byte) 0x0f,
+ });
+ assertEquals(PMTiles.readVarint(b), 9007199254740991L);
+ }
+
+ @Test
+ void zxyToTileId() {
+ assertEquals(PMTiles.zxyToTileId(0, 0, 0), 0);
+ assertEquals(PMTiles.zxyToTileId(1, 0, 0), 1);
+ assertEquals(PMTiles.zxyToTileId(1, 0, 1), 2);
+ assertEquals(PMTiles.zxyToTileId(1, 1, 1), 3);
+ assertEquals(PMTiles.zxyToTileId(1, 1, 0), 4);
+ assertEquals(PMTiles.zxyToTileId(2, 0, 0), 5);
+ }
+
+ @Test
+ void tileIdToZxy() {
+ assertArrayEquals(PMTiles.tileIdToZxy(0), new long[]{0, 0, 0});
+ assertArrayEquals(PMTiles.tileIdToZxy(1), new long[]{1, 0, 0});
+ assertArrayEquals(PMTiles.tileIdToZxy(2), new long[]{1, 0, 1});
+ assertArrayEquals(PMTiles.tileIdToZxy(3), new long[]{1, 1, 1});
+ assertArrayEquals(PMTiles.tileIdToZxy(4), new long[]{1, 1, 0});
+ assertArrayEquals(PMTiles.tileIdToZxy(5), new long[]{2, 0, 0});
+ }
+
+ @Test
+ void aLotOfTiles() {
+ for (int z = 0; z < 9; z++) {
+ for (long x = 0; x < 1 << z; x++) {
+ for (long y = 0; y < 1 << z; y++) {
+ var result = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, x, y));
+ if (result[0] != z || result[1] != x || result[2] != y) {
+ fail("roundtrip failed");
+ }
+ }
+ }
+ }
+ }
+
+ @Test
+ void tileExtremes() {
+ for (var z = 0; z < 27; z++) {
+ var dim = LongMath.pow(2, z) - 1;
+ var tl = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, 0, 0));
+ assertArrayEquals(new long[]{z, 0, 0}, tl);
+ var tr = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, dim, 0));
+ assertArrayEquals(new long[]{z, dim, 0}, tr);
+ var bl = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, 0, dim));
+ assertArrayEquals(new long[]{z, 0, dim}, bl);
+ var br = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, dim, dim));
+ assertArrayEquals(new long[]{z, dim, dim}, br);
+ }
+ }
+
+ @Test
+ void invalidTiles() {
+ assertThrows(RuntimeException.class, () -> PMTiles.tileIdToZxy(9007199254740991L));
+ assertThrows(RuntimeException.class, () -> PMTiles.zxyToTileId(27, 0, 0));
+ assertThrows(RuntimeException.class, () -> PMTiles.zxyToTileId(0, 1, 1));
+ }
+
+}
\ No newline at end of file
[incubator-baremaps] 08/08: Correctly set header and metadata
Posted by bc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bchapuis pushed a commit to branch pmtiles
in repository https://gitbox.apache.org/repos/asf/incubator-baremaps.git
commit f65b5eebd27e4535773d48e52c9855c5a7250fe8
Author: Bertil Chapuis <bc...@gmail.com>
AuthorDate: Thu Sep 21 23:07:59 2023 +0200
Correctly set header and metadata
---
.../org/apache/baremaps/tilestore/pmtiles/PMTilesWriter.java | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesWriter.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesWriter.java
index 267f6047..08ee36ab 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesWriter.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesWriter.java
@@ -71,7 +71,6 @@ public class PMTilesWriter {
var metadata = new ObjectMapper().writeValueAsBytes(metadataMap);
-
var directories = PMTiles.optimizeDirectories(entries, 16247);
long rootOffset = 127;
long rootLength = directories.root().length;
@@ -103,7 +102,13 @@ public class PMTilesWriter {
header.setLeavesLength(leavesLength);
header.setTilesOffset(tilesOffset);
header.setTilesLength(tilesLength);
-
+ header.setCenterZoom(14);
+ header.setCenterLat(46.5197);
+ header.setCenterLon(6.6323);
+ header.setMinLon(-180);
+ header.setMinLat(-90);
+ header.setMaxLon(180);
+ header.setMaxLat(90);
try (var output = new LittleEndianDataOutputStream(new FileOutputStream(path.toFile()))) {
PMTiles.serializeHeader(output, header);
[incubator-baremaps] 03/08: Format code
Posted by bc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bchapuis pushed a commit to branch pmtiles
in repository https://gitbox.apache.org/repos/asf/incubator-baremaps.git
commit 61e28f36f9fc7e70a7b16d2d444299ed96fbc3e0
Author: Bertil Chapuis <bc...@gmail.com>
AuthorDate: Wed Sep 6 00:09:17 2023 +0200
Format code
---
.../apache/baremaps/tilestore/pmtiles/PMTiles.java | 476 +++++++++++----------
.../baremaps/tilestore/pmtiles/PMTilesTest.java | 289 +++++++------
2 files changed, 393 insertions(+), 372 deletions(-)
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
index f773c7eb..3ec6d62e 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
@@ -1,261 +1,271 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
package org.apache.baremaps.tilestore.pmtiles;
import com.google.common.math.LongMath;
-
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
public class PMTiles {
- public static long toNum(long low, long high) {
- return high * 0x100000000L + low;
- }
+ public static long toNum(long low, long high) {
+ return high * 0x100000000L + low;
+ }
- public static long readVarintRemainder(long l, ByteBuffer buf) {
- long h, b;
- b = buf.get() & 0xff;
- h = (b & 0x70) >> 4;
- if (b < 0x80) {
- return toNum(l, h);
- }
- b = buf.get() & 0xff;
- h |= (b & 0x7f) << 3;
- if (b < 0x80) {
- return toNum(l, h);
- }
- b = buf.get() & 0xff;
- h |= (b & 0x7f) << 10;
- if (b < 0x80) {
- return toNum(l, h);
- }
- b = buf.get() & 0xff;
- h |= (b & 0x7f) << 17;
- if (b < 0x80) {
- return toNum(l, h);
- }
- b = buf.get() & 0xff;
- h |= (b & 0x7f) << 24;
- if (b < 0x80) {
- return toNum(l, h);
- }
- b = buf.get() & 0xff;
- h |= (b & 0x01) << 31;
- if (b < 0x80) {
- return toNum(l, h);
- }
- throw new RuntimeException("Expected varint not more than 10 bytes");
+ public static long readVarintRemainder(long l, ByteBuffer buf) {
+ long h, b;
+ b = buf.get() & 0xff;
+ h = (b & 0x70) >> 4;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ b = buf.get() & 0xff;
+ h |= (b & 0x7f) << 3;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ b = buf.get() & 0xff;
+ h |= (b & 0x7f) << 10;
+ if (b < 0x80) {
+ return toNum(l, h);
}
+ b = buf.get() & 0xff;
+ h |= (b & 0x7f) << 17;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ b = buf.get() & 0xff;
+ h |= (b & 0x7f) << 24;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ b = buf.get() & 0xff;
+ h |= (b & 0x01) << 31;
+ if (b < 0x80) {
+ return toNum(l, h);
+ }
+ throw new RuntimeException("Expected varint not more than 10 bytes");
+ }
- public static long readVarint(ByteBuffer buf) {
- long val, b;
- b = buf.get() & 0xff;
- val = b & 0x7f;
- if (b < 0x80) {
- return val;
- }
- b = buf.get() & 0xff;
- val |= (b & 0x7f) << 7;
- if (b < 0x80) {
- return val;
- }
- b = buf.get() & 0xff;
- val |= (b & 0x7f) << 14;
- if (b < 0x80) {
- return val;
- }
- b = buf.get() & 0xff;
- val |= (b & 0x7f) << 21;
- if (b < 0x80) {
- return val;
- }
- val |= (b & 0x0f) << 28;
- return readVarintRemainder(val, buf);
+ public static long readVarint(ByteBuffer buf) {
+ long val, b;
+ b = buf.get() & 0xff;
+ val = b & 0x7f;
+ if (b < 0x80) {
+ return val;
+ }
+ b = buf.get() & 0xff;
+ val |= (b & 0x7f) << 7;
+ if (b < 0x80) {
+ return val;
}
+ b = buf.get() & 0xff;
+ val |= (b & 0x7f) << 14;
+ if (b < 0x80) {
+ return val;
+ }
+ b = buf.get() & 0xff;
+ val |= (b & 0x7f) << 21;
+ if (b < 0x80) {
+ return val;
+ }
+ val |= (b & 0x0f) << 28;
+ return readVarintRemainder(val, buf);
+ }
- public static void rotate(long n, long[] xy, long rx, long ry) {
- if (ry == 0) {
- if (rx == 1) {
- xy[0] = n - 1 - xy[0];
- xy[1] = n - 1 - xy[1];
- }
- long t = xy[0];
- xy[0] = xy[1];
- xy[1] = t;
- }
+ public static void rotate(long n, long[] xy, long rx, long ry) {
+ if (ry == 0) {
+ if (rx == 1) {
+ xy[0] = n - 1 - xy[0];
+ xy[1] = n - 1 - xy[1];
+ }
+ long t = xy[0];
+ xy[0] = xy[1];
+ xy[1] = t;
}
+ }
- public static long[] idOnLevel(int z, long pos) {
- long n = LongMath.pow(2, z);
- long rx, ry, t = pos;
- long[] xy = new long[]{0, 0};
- long s = 1;
- while (s < n) {
- rx = 1 & (t / 2);
- ry = 1 & (t ^ rx);
- rotate(s, xy, rx, ry);
- xy[0] += s * rx;
- xy[1] += s * ry;
- t = t / 4;
- s *= 2;
- }
- return new long[]{z, xy[0], xy[1]};
+ public static long[] idOnLevel(int z, long pos) {
+ long n = LongMath.pow(2, z);
+ long rx, ry, t = pos;
+ long[] xy = new long[] {0, 0};
+ long s = 1;
+ while (s < n) {
+ rx = 1 & (t / 2);
+ ry = 1 & (t ^ rx);
+ rotate(s, xy, rx, ry);
+ xy[0] += s * rx;
+ xy[1] += s * ry;
+ t = t / 4;
+ s *= 2;
}
+ return new long[] {z, xy[0], xy[1]};
+ }
- private static long[] tzValues = new long[]{
- 0, 1, 5, 21, 85, 341, 1365, 5461, 21845, 87381, 349525, 1398101, 5592405,
- 22369621, 89478485, 357913941, 1431655765, 5726623061L, 22906492245L,
- 91625968981L, 366503875925L, 1466015503701L, 5864062014805L, 23456248059221L,
- 93824992236885L, 375299968947541L, 1501199875790165L,
- };
+ private static long[] tzValues = new long[] {
+ 0, 1, 5, 21, 85, 341, 1365, 5461, 21845, 87381, 349525, 1398101, 5592405,
+ 22369621, 89478485, 357913941, 1431655765, 5726623061L, 22906492245L,
+ 91625968981L, 366503875925L, 1466015503701L, 5864062014805L, 23456248059221L,
+ 93824992236885L, 375299968947541L, 1501199875790165L,
+ };
- public static long zxyToTileId(int z, long x, long y) {
- if (z > 26) {
- throw new RuntimeException("Tile zoom level exceeds max safe number limit (26)");
- }
- if (x > Math.pow(2, z) - 1 || y > Math.pow(2, z) - 1) {
- throw new RuntimeException("tile x/y outside zoom level bounds");
- }
- long acc = tzValues[z];
- long n = LongMath.pow(2, z);
- long rx = 0;
- long ry = 0;
- long d = 0;
- long[] xy = new long[]{x, y};
- long s = n / 2;
- while (s > 0) {
- rx = (xy[0] & s) > 0 ? 1 : 0;
- ry = (xy[1] & s) > 0 ? 1 : 0;
- d += s * s * ((3 * rx) ^ ry);
- rotate(s, xy, rx, ry);
- s = s / 2;
- }
- return acc + d;
+ public static long zxyToTileId(int z, long x, long y) {
+ if (z > 26) {
+ throw new RuntimeException("Tile zoom level exceeds max safe number limit (26)");
}
-
- public static long[] tileIdToZxy(long i) {
- long acc = 0;
- for (int z = 0; z < 27; z++) {
- long numTiles = (0x1L << z) * (0x1L << z);
- if (acc + numTiles > i) {
- return idOnLevel(z, i - acc);
- }
- acc += numTiles;
- }
- throw new RuntimeException("Tile zoom level exceeds max safe number limit (26)");
+ if (x > Math.pow(2, z) - 1 || y > Math.pow(2, z) - 1) {
+ throw new RuntimeException("tile x/y outside zoom level bounds");
}
-
- enum Compression {
- Unknown,
- None,
- Gzip,
- Brotli,
- Zstd,
+ long acc = tzValues[z];
+ long n = LongMath.pow(2, z);
+ long rx = 0;
+ long ry = 0;
+ long d = 0;
+ long[] xy = new long[] {x, y};
+ long s = n / 2;
+ while (s > 0) {
+ rx = (xy[0] & s) > 0 ? 1 : 0;
+ ry = (xy[1] & s) > 0 ? 1 : 0;
+ d += s * s * ((3 * rx) ^ ry);
+ rotate(s, xy, rx, ry);
+ s = s / 2;
}
+ return acc + d;
+ }
- enum TileType {
- Unknown,
- Mvt,
- Png,
- Jpeg,
- Webp,
- Avif,
+ public static long[] tileIdToZxy(long i) {
+ long acc = 0;
+ for (int z = 0; z < 27; z++) {
+ long numTiles = (0x1L << z) * (0x1L << z);
+ if (acc + numTiles > i) {
+ return idOnLevel(z, i - acc);
+ }
+ acc += numTiles;
}
+ throw new RuntimeException("Tile zoom level exceeds max safe number limit (26)");
+ }
- public record Header(
- int specVersion,
- long rootDirectoryOffset,
- long rootDirectoryLength,
- long jsonMetadataOffset,
- long jsonMetadataLength,
- long leafDirectoryOffset,
- long leafDirectoryLength,
- long tileDataOffset,
- long tileDataLength,
- long numAddressedTiles,
- long numTileEntries,
- long numTileContents,
- boolean clustered,
- Compression internalCompression,
- Compression tileCompression,
- TileType tileType,
- int minZoom,
- int maxZoom,
- double minLon,
- double minLat,
- double maxLon,
- double maxLat,
- int centerZoom,
- double centerLon,
- double centerLat,
- String etag
- ) {}
+ enum Compression {
+ Unknown,
+ None,
+ Gzip,
+ Brotli,
+ Zstd,
+ }
- public static Header bytesToHeader(ByteBuffer buf, String etag) {
- buf.order(ByteOrder.LITTLE_ENDIAN);
- return new Header(
- buf.get(7),
- buf.getLong(8),
- buf.getLong(16),
- buf.getLong(24),
- buf.getLong(32),
- buf.getLong(40),
- buf.getLong(48),
- buf.getLong(56),
- buf.getLong(64),
- buf.getLong(72),
- buf.getLong(80),
- buf.getLong(88),
- buf.get(96) == 1,
- Compression.values()[buf.get(97)],
- Compression.values()[buf.get(98)],
- TileType.values()[buf.get(99)],
- buf.get(100),
- buf.get(101),
- (double) buf.getInt(102) / 10000000,
- (double) buf.getInt(106) / 10000000,
- (double) buf.getInt(110) / 10000000,
- (double) buf.getInt(114) / 10000000,
- buf.get(118),
- (double) buf.getInt(119) / 10000000,
- (double) buf.getInt(123) / 10000000,
- etag
- );
- }
+ enum TileType {
+ Unknown,
+ Mvt,
+ Png,
+ Jpeg,
+ Webp,
+ Avif,
+ }
- public static void headerToBytes(Header header, ByteBuffer buf) {
- buf.order(ByteOrder.LITTLE_ENDIAN);
- buf.put(0, (byte) 0x4d);
- buf.put(1, (byte) 0x42);
- buf.put(2, (byte) 0x54);
- buf.put(3, (byte) 0x42);
- buf.put(4, (byte) 0x49);
- buf.put(5, (byte) 0x4e);
- buf.put(6, (byte) 0x41);
- buf.put(7, (byte) header.specVersion);
- buf.putLong(8, header.rootDirectoryOffset);
- buf.putLong(16, header.rootDirectoryLength);
- buf.putLong(24, header.jsonMetadataOffset);
- buf.putLong(32, header.jsonMetadataLength);
- buf.putLong(40, header.leafDirectoryOffset);
- buf.putLong(48, header.leafDirectoryLength);
- buf.putLong(56, header.tileDataOffset);
- buf.putLong(64, header.tileDataLength);
- buf.putLong(72, header.numAddressedTiles);
- buf.putLong(80, header.numTileEntries);
- buf.putLong(88, header.numTileContents);
- buf.put(96, (byte) (header.clustered ? 1 : 0));
- buf.put(97, (byte) header.internalCompression.ordinal());
- buf.put(98, (byte) header.tileCompression.ordinal());
- buf.put(99, (byte) header.tileType.ordinal());
- buf.put(100, (byte) header.minZoom);
- buf.put(101, (byte) header.maxZoom);
- buf.putInt(102, (int) (header.minLon * 10000000));
- buf.putInt(106, (int) (header.minLat * 10000000));
- buf.putInt(110, (int) (header.maxLon * 10000000));
- buf.putInt(114, (int) (header.maxLat * 10000000));
- buf.put(118, (byte) header.centerZoom);
- buf.putInt(119, (int) (header.centerLon * 10000000));
- buf.putInt(123, (int) (header.centerLat * 10000000));
- }
+ public record Header(
+ int specVersion,
+ long rootDirectoryOffset,
+ long rootDirectoryLength,
+ long jsonMetadataOffset,
+ long jsonMetadataLength,
+ long leafDirectoryOffset,
+ long leafDirectoryLength,
+ long tileDataOffset,
+ long tileDataLength,
+ long numAddressedTiles,
+ long numTileEntries,
+ long numTileContents,
+ boolean clustered,
+ Compression internalCompression,
+ Compression tileCompression,
+ TileType tileType,
+ int minZoom,
+ int maxZoom,
+ double minLon,
+ double minLat,
+ double maxLon,
+ double maxLat,
+ int centerZoom,
+ double centerLon,
+ double centerLat,
+ String etag) {
+ }
+
+ public static Header bytesToHeader(ByteBuffer buf, String etag) {
+ buf.order(ByteOrder.LITTLE_ENDIAN);
+ return new Header(
+ buf.get(7),
+ buf.getLong(8),
+ buf.getLong(16),
+ buf.getLong(24),
+ buf.getLong(32),
+ buf.getLong(40),
+ buf.getLong(48),
+ buf.getLong(56),
+ buf.getLong(64),
+ buf.getLong(72),
+ buf.getLong(80),
+ buf.getLong(88),
+ buf.get(96) == 1,
+ Compression.values()[buf.get(97)],
+ Compression.values()[buf.get(98)],
+ TileType.values()[buf.get(99)],
+ buf.get(100),
+ buf.get(101),
+ (double) buf.getInt(102) / 10000000,
+ (double) buf.getInt(106) / 10000000,
+ (double) buf.getInt(110) / 10000000,
+ (double) buf.getInt(114) / 10000000,
+ buf.get(118),
+ (double) buf.getInt(119) / 10000000,
+ (double) buf.getInt(123) / 10000000,
+ etag);
+ }
+
+ public static void headerToBytes(Header header, ByteBuffer buf) {
+ buf.order(ByteOrder.LITTLE_ENDIAN);
+ buf.put(0, (byte) 0x50);
+ buf.put(1, (byte) 0x4D);
+ buf.put(2, (byte) 0x54);
+ buf.put(3, (byte) 0x69);
+ buf.put(4, (byte) 0x6C);
+ buf.put(5, (byte) 0x65);
+ buf.put(6, (byte) 0x73);
+ buf.put(7, (byte) header.specVersion);
+ buf.putLong(8, header.rootDirectoryOffset);
+ buf.putLong(16, header.rootDirectoryLength);
+ buf.putLong(24, header.jsonMetadataOffset);
+ buf.putLong(32, header.jsonMetadataLength);
+ buf.putLong(40, header.leafDirectoryOffset);
+ buf.putLong(48, header.leafDirectoryLength);
+ buf.putLong(56, header.tileDataOffset);
+ buf.putLong(64, header.tileDataLength);
+ buf.putLong(72, header.numAddressedTiles);
+ buf.putLong(80, header.numTileEntries);
+ buf.putLong(88, header.numTileContents);
+ buf.put(96, (byte) (header.clustered ? 1 : 0));
+ buf.put(97, (byte) header.internalCompression.ordinal());
+ buf.put(98, (byte) header.tileCompression.ordinal());
+ buf.put(99, (byte) header.tileType.ordinal());
+ buf.put(100, (byte) header.minZoom);
+ buf.put(101, (byte) header.maxZoom);
+ buf.putInt(102, (int) (header.minLon * 10000000));
+ buf.putInt(106, (int) (header.minLat * 10000000));
+ buf.putInt(110, (int) (header.maxLon * 10000000));
+ buf.putInt(114, (int) (header.maxLat * 10000000));
+ buf.put(118, (byte) header.centerZoom);
+ buf.putInt(119, (int) (header.centerLon * 10000000));
+ buf.putInt(123, (int) (header.centerLat * 10000000));
+ }
}
diff --git a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
index 3eda409b..1fdf7129 100644
--- a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
+++ b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
@@ -1,161 +1,172 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
package org.apache.baremaps.tilestore.pmtiles;
+import static org.junit.jupiter.api.Assertions.*;
+
import com.google.common.math.LongMath;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
import org.apache.baremaps.testing.TestFiles;
import org.apache.baremaps.tilestore.pmtiles.PMTiles.Compression;
import org.apache.baremaps.tilestore.pmtiles.PMTiles.TileType;
import org.junit.jupiter.api.Test;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.file.Files;
-
-import static org.junit.jupiter.api.Assertions.*;
-
class PMTilesTest {
- @Test
- void varint() {
- var b = ByteBuffer.wrap(new byte[]{
- (byte) 0, (byte) 1,
- (byte) 127, (byte) 0xe5,
- (byte) 0x8e, (byte) 0x26
- });
- assertEquals(PMTiles.readVarint(b), 0);
- assertEquals(PMTiles.readVarint(b), 1);
- assertEquals(PMTiles.readVarint(b), 127);
- assertEquals(PMTiles.readVarint(b), 624485);
- b = ByteBuffer.wrap(new byte[]{
- (byte) 0xff, (byte) 0xff,
- (byte) 0xff, (byte) 0xff,
- (byte) 0xff, (byte) 0xff,
- (byte) 0xff, (byte) 0x0f,
- });
- assertEquals(PMTiles.readVarint(b), 9007199254740991L);
- }
+ @Test
+ void varint() {
+ var b = ByteBuffer.wrap(new byte[] {
+ (byte) 0, (byte) 1,
+ (byte) 127, (byte) 0xe5,
+ (byte) 0x8e, (byte) 0x26
+ });
+ assertEquals(PMTiles.readVarint(b), 0);
+ assertEquals(PMTiles.readVarint(b), 1);
+ assertEquals(PMTiles.readVarint(b), 127);
+ assertEquals(PMTiles.readVarint(b), 624485);
+ b = ByteBuffer.wrap(new byte[] {
+ (byte) 0xff, (byte) 0xff,
+ (byte) 0xff, (byte) 0xff,
+ (byte) 0xff, (byte) 0xff,
+ (byte) 0xff, (byte) 0x0f,
+ });
+ assertEquals(PMTiles.readVarint(b), 9007199254740991L);
+ }
- @Test
- void zxyToTileId() {
- assertEquals(PMTiles.zxyToTileId(0, 0, 0), 0);
- assertEquals(PMTiles.zxyToTileId(1, 0, 0), 1);
- assertEquals(PMTiles.zxyToTileId(1, 0, 1), 2);
- assertEquals(PMTiles.zxyToTileId(1, 1, 1), 3);
- assertEquals(PMTiles.zxyToTileId(1, 1, 0), 4);
- assertEquals(PMTiles.zxyToTileId(2, 0, 0), 5);
- }
+ @Test
+ void zxyToTileId() {
+ assertEquals(PMTiles.zxyToTileId(0, 0, 0), 0);
+ assertEquals(PMTiles.zxyToTileId(1, 0, 0), 1);
+ assertEquals(PMTiles.zxyToTileId(1, 0, 1), 2);
+ assertEquals(PMTiles.zxyToTileId(1, 1, 1), 3);
+ assertEquals(PMTiles.zxyToTileId(1, 1, 0), 4);
+ assertEquals(PMTiles.zxyToTileId(2, 0, 0), 5);
+ }
- @Test
- void tileIdToZxy() {
- assertArrayEquals(PMTiles.tileIdToZxy(0), new long[]{0, 0, 0});
- assertArrayEquals(PMTiles.tileIdToZxy(1), new long[]{1, 0, 0});
- assertArrayEquals(PMTiles.tileIdToZxy(2), new long[]{1, 0, 1});
- assertArrayEquals(PMTiles.tileIdToZxy(3), new long[]{1, 1, 1});
- assertArrayEquals(PMTiles.tileIdToZxy(4), new long[]{1, 1, 0});
- assertArrayEquals(PMTiles.tileIdToZxy(5), new long[]{2, 0, 0});
- }
+ @Test
+ void tileIdToZxy() {
+ assertArrayEquals(PMTiles.tileIdToZxy(0), new long[] {0, 0, 0});
+ assertArrayEquals(PMTiles.tileIdToZxy(1), new long[] {1, 0, 0});
+ assertArrayEquals(PMTiles.tileIdToZxy(2), new long[] {1, 0, 1});
+ assertArrayEquals(PMTiles.tileIdToZxy(3), new long[] {1, 1, 1});
+ assertArrayEquals(PMTiles.tileIdToZxy(4), new long[] {1, 1, 0});
+ assertArrayEquals(PMTiles.tileIdToZxy(5), new long[] {2, 0, 0});
+ }
- @Test
- void aLotOfTiles() {
- for (int z = 0; z < 9; z++) {
- for (long x = 0; x < 1 << z; x++) {
- for (long y = 0; y < 1 << z; y++) {
- var result = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, x, y));
- if (result[0] != z || result[1] != x || result[2] != y) {
- fail("roundtrip failed");
- }
- }
- }
+ @Test
+ void aLotOfTiles() {
+ for (int z = 0; z < 9; z++) {
+ for (long x = 0; x < 1 << z; x++) {
+ for (long y = 0; y < 1 << z; y++) {
+ var result = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, x, y));
+ if (result[0] != z || result[1] != x || result[2] != y) {
+ fail("roundtrip failed");
+ }
}
+ }
}
+ }
- @Test
- void tileExtremes() {
- for (var z = 0; z < 27; z++) {
- var dim = LongMath.pow(2, z) - 1;
- var tl = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, 0, 0));
- assertArrayEquals(new long[]{z, 0, 0}, tl);
- var tr = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, dim, 0));
- assertArrayEquals(new long[]{z, dim, 0}, tr);
- var bl = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, 0, dim));
- assertArrayEquals(new long[]{z, 0, dim}, bl);
- var br = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, dim, dim));
- assertArrayEquals(new long[]{z, dim, dim}, br);
- }
+ @Test
+ void tileExtremes() {
+ for (var z = 0; z < 27; z++) {
+ var dim = LongMath.pow(2, z) - 1;
+ var tl = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, 0, 0));
+ assertArrayEquals(new long[] {z, 0, 0}, tl);
+ var tr = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, dim, 0));
+ assertArrayEquals(new long[] {z, dim, 0}, tr);
+ var bl = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, 0, dim));
+ assertArrayEquals(new long[] {z, 0, dim}, bl);
+ var br = PMTiles.tileIdToZxy(PMTiles.zxyToTileId(z, dim, dim));
+ assertArrayEquals(new long[] {z, dim, dim}, br);
}
+ }
- @Test
- void invalidTiles() {
- assertThrows(RuntimeException.class, () -> PMTiles.tileIdToZxy(9007199254740991L));
- assertThrows(RuntimeException.class, () -> PMTiles.zxyToTileId(27, 0, 0));
- assertThrows(RuntimeException.class, () -> PMTiles.zxyToTileId(0, 1, 1));
- }
+ @Test
+ void invalidTiles() {
+ assertThrows(RuntimeException.class, () -> PMTiles.tileIdToZxy(9007199254740991L));
+ assertThrows(RuntimeException.class, () -> PMTiles.zxyToTileId(27, 0, 0));
+ assertThrows(RuntimeException.class, () -> PMTiles.zxyToTileId(0, 1, 1));
+ }
- @Test
- void bytesToHeader() throws IOException {
- var file = TestFiles.resolve("pmtiles/test_fixture_1.pmtiles");
- try (var channel = Files.newByteChannel(file)) {
- var buffer = ByteBuffer.allocate(127);
- channel.read(buffer);
- buffer.flip();
- var header = PMTiles.bytesToHeader(buffer, "1");
- assertEquals(header.rootDirectoryOffset(), 127);
- assertEquals(header.rootDirectoryLength(), 25);
- assertEquals(header.jsonMetadataOffset(), 152);
- assertEquals(header.jsonMetadataLength(), 247);
- assertEquals(header.leafDirectoryOffset(), 0);
- assertEquals(header.leafDirectoryLength(), 0);
- assertEquals(header.tileDataOffset(), 399);
- assertEquals(header.tileDataLength(), 69);
- assertEquals(header.numAddressedTiles(), 1);
- assertEquals(header.numTileEntries(), 1);
- assertEquals(header.numTileContents(), 1);
- assertFalse(header.clustered());
- assertEquals(header.internalCompression(), Compression.Gzip);
- assertEquals(header.tileCompression(), Compression.Gzip);
- assertEquals(header.tileType(), TileType.Mvt);
- assertEquals(header.minZoom(), 0);
- assertEquals(header.maxZoom(), 0);
- assertEquals(header.minLon(), 0);
- assertEquals(header.minLat(), 0);
- assertEquals(Math.round(header.maxLon()), 1);
- assertEquals(Math.round(header.maxLat()), 1);
- }
+ @Test
+ void bytesToHeader() throws IOException {
+ var file = TestFiles.resolve("pmtiles/test_fixture_1.pmtiles");
+ try (var channel = Files.newByteChannel(file)) {
+ var buffer = ByteBuffer.allocate(127);
+ channel.read(buffer);
+ buffer.flip();
+ var header = PMTiles.bytesToHeader(buffer, "1");
+ assertEquals(header.rootDirectoryOffset(), 127);
+ assertEquals(header.rootDirectoryLength(), 25);
+ assertEquals(header.jsonMetadataOffset(), 152);
+ assertEquals(header.jsonMetadataLength(), 247);
+ assertEquals(header.leafDirectoryOffset(), 0);
+ assertEquals(header.leafDirectoryLength(), 0);
+ assertEquals(header.tileDataOffset(), 399);
+ assertEquals(header.tileDataLength(), 69);
+ assertEquals(header.numAddressedTiles(), 1);
+ assertEquals(header.numTileEntries(), 1);
+ assertEquals(header.numTileContents(), 1);
+ assertFalse(header.clustered());
+ assertEquals(header.internalCompression(), Compression.Gzip);
+ assertEquals(header.tileCompression(), Compression.Gzip);
+ assertEquals(header.tileType(), TileType.Mvt);
+ assertEquals(header.minZoom(), 0);
+ assertEquals(header.maxZoom(), 0);
+ assertEquals(header.minLon(), 0);
+ assertEquals(header.minLat(), 0);
+ assertEquals(Math.round(header.maxLon()), 1);
+ assertEquals(Math.round(header.maxLat()), 1);
}
+ }
- @Test
- void headerToBytes() throws IOException {
- var etag = "1";
- var buffer = ByteBuffer.allocate(127);
- var header = new PMTiles.Header(
- 127,
- 25,
- 152,
- 247,
- 0,
- 0,
- 399,
- 69,
- 1,
- 1,
- 1,
- 10,
- false,
- Compression.Gzip,
- Compression.Gzip,
- TileType.Mvt,
- 0,
- 0,
- 0,
- 1,
- 1,
- 0,
- 0,
- 0,
- 0,
- etag);
- PMTiles.headerToBytes(header, buffer);
- var header2 = PMTiles.bytesToHeader(buffer, etag);
- assertEquals(header, header2);
- }
+ @Test
+ void headerToBytes() throws IOException {
+ var etag = "1";
+ var buffer = ByteBuffer.allocate(127);
+ var header = new PMTiles.Header(
+ 127,
+ 25,
+ 152,
+ 247,
+ 0,
+ 0,
+ 399,
+ 69,
+ 1,
+ 1,
+ 1,
+ 10,
+ false,
+ Compression.Gzip,
+ Compression.Gzip,
+ TileType.Mvt,
+ 0,
+ 0,
+ 0,
+ 1,
+ 1,
+ 0,
+ 0,
+ 0,
+ 0,
+ etag);
+ PMTiles.headerToBytes(header, buffer);
+ var header2 = PMTiles.bytesToHeader(buffer, etag);
+ assertEquals(header, header2);
+ }
-}
\ No newline at end of file
+}
[incubator-baremaps] 07/08: Add a minimalistic tile writer
Posted by bc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bchapuis pushed a commit to branch pmtiles
in repository https://gitbox.apache.org/repos/asf/incubator-baremaps.git
commit 4633de7ba88c0cad42f08050c39d47139e904487
Author: Bertil Chapuis <bc...@gmail.com>
AuthorDate: Mon Sep 18 18:46:10 2023 +0200
Add a minimalistic tile writer
---
.../org/apache/baremaps/tilestore/TileCache.java | 6 +
.../org/apache/baremaps/tilestore/TileStore.java | 2 +-
.../baremaps/tilestore/file/FileTileStore.java | 5 +
.../baremaps/tilestore/mbtiles/MBTilesStore.java | 5 +
.../baremaps/tilestore/pmtiles/Compression.java | 109 ++--
.../apache/baremaps/tilestore/pmtiles/Entry.java | 104 ++--
.../apache/baremaps/tilestore/pmtiles/Header.java | 556 ++++++++++++---------
.../apache/baremaps/tilestore/pmtiles/PMTiles.java | 263 ++++++----
.../baremaps/tilestore/pmtiles/PMTilesReader.java | 91 ++++
.../PMTilesStore.java} | 48 +-
.../baremaps/tilestore/pmtiles/PMTilesWriter.java | 119 +++++
.../baremaps/tilestore/pmtiles/TileType.java | 24 +-
.../tilestore/postgres/PostgresTileStore.java | 4 +
.../baremaps/workflow/tasks/ExportVectorTiles.java | 34 +-
.../baremaps/tilestore/pmtiles/PMTilesReader.java | 77 ---
.../baremaps/tilestore/pmtiles/PMTilesTest.java | 125 ++---
16 files changed, 939 insertions(+), 633 deletions(-)
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/TileCache.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/TileCache.java
index 6e037790..6d73135a 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/TileCache.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/TileCache.java
@@ -79,4 +79,10 @@ public class TileCache implements TileStore {
tileStore.delete(tileCoord);
cache.invalidate(tileCoord);
}
+
+ @Override
+ public void close() throws Exception {
+ tileStore.close();
+ cache.cleanUp();
+ }
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/TileStore.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/TileStore.java
index c7288383..11b66a9c 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/TileStore.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/TileStore.java
@@ -17,7 +17,7 @@ package org.apache.baremaps.tilestore;
import java.nio.ByteBuffer;
/** Represents a store for tiles. */
-public interface TileStore {
+public interface TileStore extends AutoCloseable {
/**
* Reads the content of a tile.
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/file/FileTileStore.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/file/FileTileStore.java
index 77424004..0cba2837 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/file/FileTileStore.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/file/FileTileStore.java
@@ -72,4 +72,9 @@ public class FileTileStore implements TileStore {
public Path resolve(TileCoord tileCoord) {
return path.resolve(String.format("%s/%s/%s.mvt", tileCoord.z(), tileCoord.x(), tileCoord.y()));
}
+
+ @Override
+ public void close() {
+ // Do nothing
+ }
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/mbtiles/MBTilesStore.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/mbtiles/MBTilesStore.java
index a5cb03e0..d21cf07d 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/mbtiles/MBTilesStore.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/mbtiles/MBTilesStore.java
@@ -183,4 +183,9 @@ public class MBTilesStore implements TileStore {
private static int reverseY(int y, int z) {
return (int) (Math.pow(2.0, z) - 1 - y);
}
+
+ @Override
+ public void close() throws Exception {
+ // Do nothing
+ }
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Compression.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Compression.java
index 4360d9c7..90613fef 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Compression.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Compression.java
@@ -1,70 +1,69 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
package org.apache.baremaps.tilestore.pmtiles;
-import java.io.ByteArrayInputStream;
-import java.io.ByteArrayOutputStream;
-import java.io.IOException;
-import java.nio.ByteBuffer;
+import java.io.*;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
enum Compression {
- Unknown,
- None,
- Gzip,
- Brotli,
- Zstd;
+ Unknown,
+ None,
+ Gzip,
+ Brotli,
+ Zstd;
- ByteBuffer decompress(ByteBuffer buffer) {
- return switch (this) {
- case None -> buffer;
- case Gzip -> decompressGzip(buffer);
- case Brotli -> decompressBrotli(buffer);
- case Zstd -> decompressZstd(buffer);
- default -> throw new RuntimeException("Unknown compression");
- };
- }
+ InputStream decompress(InputStream inputStream) throws IOException {
+ return switch (this) {
+ case None -> inputStream;
+ case Gzip -> decompressGzip(inputStream);
+ case Brotli -> decompressBrotli(inputStream);
+ case Zstd -> decompressZstd(inputStream);
+ default -> throw new RuntimeException("Unknown compression");
+ };
+ }
- static ByteBuffer decompressGzip(ByteBuffer buffer) {
- try(var inputStream = new GZIPInputStream(new ByteArrayInputStream(buffer.array()))) {
- return ByteBuffer.wrap(inputStream.readAllBytes());
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
+ static InputStream decompressGzip(InputStream inputStream) throws IOException {
+ return new GZIPInputStream(inputStream);
+ }
- static ByteBuffer decompressBrotli(ByteBuffer buffer) {
- throw new RuntimeException("Brotli compression not implemented");
- }
+ static InputStream decompressBrotli(InputStream buffer) {
+ throw new RuntimeException("Brotli compression not implemented");
+ }
- static ByteBuffer decompressZstd(ByteBuffer buffer) {
- throw new RuntimeException("Zstd compression not implemented");
- }
+ static InputStream decompressZstd(InputStream buffer) {
+ throw new RuntimeException("Zstd compression not implemented");
+ }
- ByteBuffer compress(ByteBuffer buffer) {
- return switch (this) {
- case None -> buffer;
- case Gzip -> compressGzip(buffer);
- case Brotli -> compressBrotli(buffer);
- case Zstd -> compressZstd(buffer);
- default -> throw new RuntimeException("Unknown compression");
- };
- }
+ OutputStream compress(OutputStream outputStream) throws IOException {
+ return switch (this) {
+ case None -> outputStream;
+ case Gzip -> compressGzip(outputStream);
+ case Brotli -> compressBrotli(outputStream);
+ case Zstd -> compressZstd(outputStream);
+ default -> throw new RuntimeException("Unknown compression");
+ };
+ }
- static ByteBuffer compressGzip(ByteBuffer buffer) {
- try(var outputStream = new ByteArrayOutputStream();
- var gzipOutputStream = new GZIPOutputStream(outputStream)) {
- gzipOutputStream.write(buffer.array());
- return ByteBuffer.wrap(outputStream.toByteArray());
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
+ static OutputStream compressGzip(OutputStream outputStream) throws IOException {
+ return new GZIPOutputStream(outputStream);
+ }
- static ByteBuffer compressBrotli(ByteBuffer buffer) {
- throw new RuntimeException("Brotli compression not implemented");
- }
+ static OutputStream compressBrotli(OutputStream outputStream) {
+ throw new RuntimeException("Brotli compression not implemented");
+ }
- static ByteBuffer compressZstd(ByteBuffer buffer) {
- throw new RuntimeException("Zstd compression not implemented");
- }
+ static OutputStream compressZstd(OutputStream outputStream) {
+ throw new RuntimeException("Zstd compression not implemented");
+ }
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Entry.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Entry.java
index 97fe00e2..7d194df4 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Entry.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Entry.java
@@ -1,51 +1,61 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
package org.apache.baremaps.tilestore.pmtiles;
public class Entry {
- private long tileId;
- private long offset;
- private long length;
- private long runLength;
-
- public Entry() {
-
- }
-
- public Entry(long tileId, long offset, long length, long runLength) {
- this.tileId = tileId;
- this.offset = offset;
- this.length = length;
- this.runLength = runLength;
- }
-
- public long getTileId() {
- return tileId;
- }
-
- public void setTileId(long tileId) {
- this.tileId = tileId;
- }
-
- public long getOffset() {
- return offset;
- }
-
- public void setOffset(long offset) {
- this.offset = offset;
- }
-
- public long getLength() {
- return length;
- }
-
- public void setLength(long length) {
- this.length = length;
- }
-
- public long getRunLength() {
- return runLength;
- }
-
- public void setRunLength(long runLength) {
- this.runLength = runLength;
- }
+ private long tileId;
+ private long offset;
+ private long length;
+ private long runLength;
+
+ public Entry() {}
+
+ public Entry(long tileId, long offset, long length, long runLength) {
+ this.tileId = tileId;
+ this.offset = offset;
+ this.length = length;
+ this.runLength = runLength;
+ }
+
+ public long getTileId() {
+ return tileId;
+ }
+
+ public void setTileId(long tileId) {
+ this.tileId = tileId;
+ }
+
+ public long getOffset() {
+ return offset;
+ }
+
+ public void setOffset(long offset) {
+ this.offset = offset;
+ }
+
+ public long getLength() {
+ return length;
+ }
+
+ public void setLength(long length) {
+ this.length = length;
+ }
+
+ public long getRunLength() {
+ return runLength;
+ }
+
+ public void setRunLength(long runLength) {
+ this.runLength = runLength;
+ }
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Header.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Header.java
index d3cb42cb..8497d3a9 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Header.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/Header.java
@@ -1,258 +1,318 @@
-package org.apache.baremaps.tilestore.pmtiles;
-
-public class Header {
-
- private int specVersion;
- private long rootDirectoryOffset;
- private long rootDirectoryLength;
- private long jsonMetadataOffset;
- private long jsonMetadataLength;
- private long leafDirectoryOffset;
- private long leafDirectoryLength;
- private long tileDataOffset;
- private long tileDataLength;
- private long numAddressedTiles;
- private long numTileEntries;
- private long numTileContents;
- private boolean clustered;
- private Compression internalCompression;
- private Compression tileCompression;
- private TileType tileType;
- private int minZoom;
- private int maxZoom;
- private double minLon;
- private double minLat;
- private double maxLon;
- private double maxLat;
- private int centerZoom;
- private double centerLon;
- private double centerLat;
-
- public Header(int specVersion, long rootDirectoryOffset, long rootDirectoryLength, long jsonMetadataOffset, long jsonMetadataLength, long leafDirectoryOffset, long leafDirectoryLength, long tileDataOffset, long tileDataLength, long numAddressedTiles, long numTileEntries, long numTileContents, boolean clustered, Compression internalCompression, Compression tileCompression, TileType tileType, int minZoom, int maxZoom, double minLon, double minLat, double maxLon, double maxLat, int cent [...]
- this.specVersion = specVersion;
- this.rootDirectoryOffset = rootDirectoryOffset;
- this.rootDirectoryLength = rootDirectoryLength;
- this.jsonMetadataOffset = jsonMetadataOffset;
- this.jsonMetadataLength = jsonMetadataLength;
- this.leafDirectoryOffset = leafDirectoryOffset;
- this.leafDirectoryLength = leafDirectoryLength;
- this.tileDataOffset = tileDataOffset;
- this.tileDataLength = tileDataLength;
- this.numAddressedTiles = numAddressedTiles;
- this.numTileEntries = numTileEntries;
- this.numTileContents = numTileContents;
- this.clustered = clustered;
- this.internalCompression = internalCompression;
- this.tileCompression = tileCompression;
- this.tileType = tileType;
- this.minZoom = minZoom;
- this.maxZoom = maxZoom;
- this.minLon = minLon;
- this.minLat = minLat;
- this.maxLon = maxLon;
- this.maxLat = maxLat;
- this.centerZoom = centerZoom;
- this.centerLon = centerLon;
- this.centerLat = centerLat;
- }
-
- public int getSpecVersion() {
- return specVersion;
- }
-
- public void setSpecVersion(int specVersion) {
- this.specVersion = specVersion;
- }
-
- public long getRootDirectoryOffset() {
- return rootDirectoryOffset;
- }
-
- public void setRootDirectoryOffset(long rootDirectoryOffset) {
- this.rootDirectoryOffset = rootDirectoryOffset;
- }
-
- public long getRootDirectoryLength() {
- return rootDirectoryLength;
- }
-
- public void setRootDirectoryLength(long rootDirectoryLength) {
- this.rootDirectoryLength = rootDirectoryLength;
- }
-
- public long getJsonMetadataOffset() {
- return jsonMetadataOffset;
- }
-
- public void setJsonMetadataOffset(long jsonMetadataOffset) {
- this.jsonMetadataOffset = jsonMetadataOffset;
- }
-
- public long getJsonMetadataLength() {
- return jsonMetadataLength;
- }
-
- public void setJsonMetadataLength(long jsonMetadataLength) {
- this.jsonMetadataLength = jsonMetadataLength;
- }
-
- public long getLeafDirectoryOffset() {
- return leafDirectoryOffset;
- }
-
- public void setLeafDirectoryOffset(long leafDirectoryOffset) {
- this.leafDirectoryOffset = leafDirectoryOffset;
- }
-
- public long getLeafDirectoryLength() {
- return leafDirectoryLength;
- }
-
- public void setLeafDirectoryLength(long leafDirectoryLength) {
- this.leafDirectoryLength = leafDirectoryLength;
- }
-
- public long getTileDataOffset() {
- return tileDataOffset;
- }
-
- public void setTileDataOffset(long tileDataOffset) {
- this.tileDataOffset = tileDataOffset;
- }
-
- public long getTileDataLength() {
- return tileDataLength;
- }
-
- public void setTileDataLength(long tileDataLength) {
- this.tileDataLength = tileDataLength;
- }
-
- public long getNumAddressedTiles() {
- return numAddressedTiles;
- }
-
- public void setNumAddressedTiles(long numAddressedTiles) {
- this.numAddressedTiles = numAddressedTiles;
- }
-
- public long getNumTileEntries() {
- return numTileEntries;
- }
-
- public void setNumTileEntries(long numTileEntries) {
- this.numTileEntries = numTileEntries;
- }
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
- public long getNumTileContents() {
- return numTileContents;
- }
-
- public void setNumTileContents(long numTileContents) {
- this.numTileContents = numTileContents;
- }
-
- public boolean isClustered() {
- return clustered;
- }
-
- public void setClustered(boolean clustered) {
- this.clustered = clustered;
- }
-
- public Compression getInternalCompression() {
- return internalCompression;
- }
-
- public void setInternalCompression(Compression internalCompression) {
- this.internalCompression = internalCompression;
- }
-
- public Compression getTileCompression() {
- return tileCompression;
- }
-
- public void setTileCompression(Compression tileCompression) {
- this.tileCompression = tileCompression;
- }
-
- public TileType getTileType() {
- return tileType;
- }
-
- public void setTileType(TileType tileType) {
- this.tileType = tileType;
- }
-
- public int getMinZoom() {
- return minZoom;
- }
-
- public void setMinZoom(int minZoom) {
- this.minZoom = minZoom;
- }
-
- public int getMaxZoom() {
- return maxZoom;
- }
-
- public void setMaxZoom(int maxZoom) {
- this.maxZoom = maxZoom;
- }
-
- public double getMinLon() {
- return minLon;
- }
-
- public void setMinLon(double minLon) {
- this.minLon = minLon;
- }
-
- public double getMinLat() {
- return minLat;
- }
-
- public void setMinLat(double minLat) {
- this.minLat = minLat;
- }
-
- public double getMaxLon() {
- return maxLon;
- }
-
- public void setMaxLon(double maxLon) {
- this.maxLon = maxLon;
- }
+package org.apache.baremaps.tilestore.pmtiles;
- public double getMaxLat() {
- return maxLat;
- }
+import java.util.Objects;
- public void setMaxLat(double maxLat) {
- this.maxLat = maxLat;
- }
+public class Header {
- public int getCenterZoom() {
- return centerZoom;
- }
+ private int specVersion;
+ private long rootDirectoryOffset;
+ private long rootDirectoryLength;
+ private long jsonMetadataOffset;
+ private long jsonMetadataLength;
+ private long leafDirectoryOffset;
+ private long leafDirectoryLength;
+ private long tileDataOffset;
+ private long tileDataLength;
+ private long numAddressedTiles;
+ private long numTileEntries;
+ private long numTileContents;
+ private boolean clustered;
+ private Compression internalCompression;
+ private Compression tileCompression;
+ private TileType tileType;
+ private int minZoom;
+ private int maxZoom;
+ private double minLon;
+ private double minLat;
+ private double maxLon;
+ private double maxLat;
+ private int centerZoom;
+ private double centerLon;
+ private double centerLat;
+
+ public Header() {
+ this.specVersion = 3;
+ }
+
+ public Header(int specVersion, long rootDirectoryOffset, long rootDirectoryLength,
+ long jsonMetadataOffset, long jsonMetadataLength, long leafDirectoryOffset,
+ long leafDirectoryLength, long tileDataOffset, long tileDataLength, long numAddressedTiles,
+ long numTileEntries, long numTileContents, boolean clustered, Compression internalCompression,
+ Compression tileCompression, TileType tileType, int minZoom, int maxZoom, double minLon,
+ double minLat, double maxLon, double maxLat, int centerZoom, double centerLon,
+ double centerLat) {
+ this.specVersion = specVersion;
+ this.rootDirectoryOffset = rootDirectoryOffset;
+ this.rootDirectoryLength = rootDirectoryLength;
+ this.jsonMetadataOffset = jsonMetadataOffset;
+ this.jsonMetadataLength = jsonMetadataLength;
+ this.leafDirectoryOffset = leafDirectoryOffset;
+ this.leafDirectoryLength = leafDirectoryLength;
+ this.tileDataOffset = tileDataOffset;
+ this.tileDataLength = tileDataLength;
+ this.numAddressedTiles = numAddressedTiles;
+ this.numTileEntries = numTileEntries;
+ this.numTileContents = numTileContents;
+ this.clustered = clustered;
+ this.internalCompression = internalCompression;
+ this.tileCompression = tileCompression;
+ this.tileType = tileType;
+ this.minZoom = minZoom;
+ this.maxZoom = maxZoom;
+ this.minLon = minLon;
+ this.minLat = minLat;
+ this.maxLon = maxLon;
+ this.maxLat = maxLat;
+ this.centerZoom = centerZoom;
+ this.centerLon = centerLon;
+ this.centerLat = centerLat;
+ }
+
+ public int getSpecVersion() {
+ return specVersion;
+ }
+
+ public void setSpecVersion(int specVersion) {
+ this.specVersion = specVersion;
+ }
+
+ public long getRootDirectoryOffset() {
+ return rootDirectoryOffset;
+ }
+
+ public void setRootOffset(long rootDirectoryOffset) {
+ this.rootDirectoryOffset = rootDirectoryOffset;
+ }
+
+ public long getRootDirectoryLength() {
+ return rootDirectoryLength;
+ }
+
+ public void setRootLength(long rootDirectoryLength) {
+ this.rootDirectoryLength = rootDirectoryLength;
+ }
+
+ public long getJsonMetadataOffset() {
+ return jsonMetadataOffset;
+ }
+
+ public void setMetadataOffset(long jsonMetadataOffset) {
+ this.jsonMetadataOffset = jsonMetadataOffset;
+ }
+
+ public long getJsonMetadataLength() {
+ return jsonMetadataLength;
+ }
+
+ public void setMetadataLength(long jsonMetadataLength) {
+ this.jsonMetadataLength = jsonMetadataLength;
+ }
+
+ public long getLeafDirectoryOffset() {
+ return leafDirectoryOffset;
+ }
+
+ public void setLeavesOffset(long leafDirectoryOffset) {
+ this.leafDirectoryOffset = leafDirectoryOffset;
+ }
+
+ public long getLeafDirectoryLength() {
+ return leafDirectoryLength;
+ }
+
+ public void setLeavesLength(long leafDirectoryLength) {
+ this.leafDirectoryLength = leafDirectoryLength;
+ }
+
+ public long getTileDataOffset() {
+ return tileDataOffset;
+ }
+
+ public void setTilesOffset(long tileDataOffset) {
+ this.tileDataOffset = tileDataOffset;
+ }
+
+ public long getTileDataLength() {
+ return tileDataLength;
+ }
+
+ public void setTilesLength(long tileDataLength) {
+ this.tileDataLength = tileDataLength;
+ }
+
+ public long getNumAddressedTiles() {
+ return numAddressedTiles;
+ }
+
+ public void setNumAddressedTiles(long numAddressedTiles) {
+ this.numAddressedTiles = numAddressedTiles;
+ }
+
+ public long getNumTileEntries() {
+ return numTileEntries;
+ }
+
+ public void setNumTileEntries(long numTileEntries) {
+ this.numTileEntries = numTileEntries;
+ }
+
+ public long getNumTileContents() {
+ return numTileContents;
+ }
+
+ public void setNumTileContents(long numTileContents) {
+ this.numTileContents = numTileContents;
+ }
- public void setCenterZoom(int centerZoom) {
- this.centerZoom = centerZoom;
- }
+ public boolean isClustered() {
+ return clustered;
+ }
- public double getCenterLon() {
- return centerLon;
- }
+ public void setClustered(boolean clustered) {
+ this.clustered = clustered;
+ }
- public void setCenterLon(double centerLon) {
- this.centerLon = centerLon;
- }
+ public Compression getInternalCompression() {
+ return internalCompression;
+ }
- public double getCenterLat() {
- return centerLat;
- }
+ public void setInternalCompression(Compression internalCompression) {
+ this.internalCompression = internalCompression;
+ }
- public void setCenterLat(double centerLat) {
- this.centerLat = centerLat;
- }
+ public Compression getTileCompression() {
+ return tileCompression;
+ }
+
+ public void setTileCompression(Compression tileCompression) {
+ this.tileCompression = tileCompression;
+ }
+
+ public TileType getTileType() {
+ return tileType;
+ }
+
+ public void setTileType(TileType tileType) {
+ this.tileType = tileType;
+ }
+
+ public int getMinZoom() {
+ return minZoom;
+ }
+
+ public void setMinZoom(int minZoom) {
+ this.minZoom = minZoom;
+ }
+
+ public int getMaxZoom() {
+ return maxZoom;
+ }
+
+ public void setMaxZoom(int maxZoom) {
+ this.maxZoom = maxZoom;
+ }
+
+ public double getMinLon() {
+ return minLon;
+ }
+
+ public void setMinLon(double minLon) {
+ this.minLon = minLon;
+ }
+
+ public double getMinLat() {
+ return minLat;
+ }
+
+ public void setMinLat(double minLat) {
+ this.minLat = minLat;
+ }
+
+ public double getMaxLon() {
+ return maxLon;
+ }
+
+ public void setMaxLon(double maxLon) {
+ this.maxLon = maxLon;
+ }
+
+ public double getMaxLat() {
+ return maxLat;
+ }
+
+ public void setMaxLat(double maxLat) {
+ this.maxLat = maxLat;
+ }
+
+ public int getCenterZoom() {
+ return centerZoom;
+ }
+
+ public void setCenterZoom(int centerZoom) {
+ this.centerZoom = centerZoom;
+ }
+
+ public double getCenterLon() {
+ return centerLon;
+ }
+
+ public void setCenterLon(double centerLon) {
+ this.centerLon = centerLon;
+ }
+
+ public double getCenterLat() {
+ return centerLat;
+ }
+
+ public void setCenterLat(double centerLat) {
+ this.centerLat = centerLat;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ Header header = (Header) o;
+ return specVersion == header.specVersion && rootDirectoryOffset == header.rootDirectoryOffset
+ && rootDirectoryLength == header.rootDirectoryLength
+ && jsonMetadataOffset == header.jsonMetadataOffset
+ && jsonMetadataLength == header.jsonMetadataLength
+ && leafDirectoryOffset == header.leafDirectoryOffset
+ && leafDirectoryLength == header.leafDirectoryLength
+ && tileDataOffset == header.tileDataOffset && tileDataLength == header.tileDataLength
+ && numAddressedTiles == header.numAddressedTiles && numTileEntries == header.numTileEntries
+ && numTileContents == header.numTileContents && clustered == header.clustered
+ && minZoom == header.minZoom && maxZoom == header.maxZoom
+ && Double.compare(header.minLon, minLon) == 0 && Double.compare(header.minLat, minLat) == 0
+ && Double.compare(header.maxLon, maxLon) == 0 && Double.compare(header.maxLat, maxLat) == 0
+ && centerZoom == header.centerZoom && Double.compare(header.centerLon, centerLon) == 0
+ && Double.compare(header.centerLat, centerLat) == 0
+ && internalCompression == header.internalCompression
+ && tileCompression == header.tileCompression && tileType == header.tileType;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(specVersion, rootDirectoryOffset, rootDirectoryLength, jsonMetadataOffset,
+ jsonMetadataLength, leafDirectoryOffset, leafDirectoryLength, tileDataOffset,
+ tileDataLength, numAddressedTiles, numTileEntries, numTileContents, clustered,
+ internalCompression, tileCompression, tileType, minZoom, maxZoom, minLon, minLat, maxLon,
+ maxLat, centerZoom, centerLon, centerLat);
+ }
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
index 7d260a5b..b81bf7cb 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
@@ -12,9 +12,11 @@
package org.apache.baremaps.tilestore.pmtiles;
+import com.google.common.io.LittleEndianDataInputStream;
+import com.google.common.io.LittleEndianDataOutputStream;
import com.google.common.math.LongMath;
-import java.nio.ByteBuffer;
-import java.nio.ByteOrder;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
@@ -24,34 +26,35 @@ public class PMTiles {
return high * 0x100000000L + low;
}
- public static long readVarIntRemainder(long l, ByteBuffer buf) {
+ public static long readVarIntRemainder(LittleEndianDataInputStream input, long l)
+ throws IOException {
long h, b;
- b = buf.get() & 0xff;
+ b = input.readByte() & 0xff;
h = (b & 0x70) >> 4;
if (b < 0x80) {
return toNum(l, h);
}
- b = buf.get() & 0xff;
+ b = input.readByte() & 0xff;
h |= (b & 0x7f) << 3;
if (b < 0x80) {
return toNum(l, h);
}
- b = buf.get() & 0xff;
+ b = input.readByte() & 0xff;
h |= (b & 0x7f) << 10;
if (b < 0x80) {
return toNum(l, h);
}
- b = buf.get() & 0xff;
+ b = input.readByte() & 0xff;
h |= (b & 0x7f) << 17;
if (b < 0x80) {
return toNum(l, h);
}
- b = buf.get() & 0xff;
+ b = input.readByte() & 0xff;
h |= (b & 0x7f) << 24;
if (b < 0x80) {
return toNum(l, h);
}
- b = buf.get() & 0xff;
+ b = input.readByte() & 0xff;
h |= (b & 0x01) << 31;
if (b < 0x80) {
return toNum(l, h);
@@ -59,41 +62,42 @@ public class PMTiles {
throw new RuntimeException("Expected varint not more than 10 bytes");
}
- public static int encodeVarInt(ByteBuffer buf, long value) {
+ public static int writeVarInt(LittleEndianDataOutputStream output, long value)
+ throws IOException {
int n = 1;
while (value >= 0x80) {
- buf.put((byte) (value | 0x80));
+ output.writeByte((byte) (value | 0x80));
value >>>= 7;
n++;
}
- buf.put((byte) value);
+ output.writeByte((byte) value);
return n;
}
- public static long decodeVarInt(ByteBuffer buf) {
+ public static long readVarInt(LittleEndianDataInputStream input) throws IOException {
long val, b;
- b = buf.get() & 0xff;
+ b = input.readByte() & 0xff;
val = b & 0x7f;
if (b < 0x80) {
return val;
}
- b = buf.get() & 0xff;
+ b = input.readByte() & 0xff;
val |= (b & 0x7f) << 7;
if (b < 0x80) {
return val;
}
- b = buf.get() & 0xff;
+ b = input.readByte() & 0xff;
val |= (b & 0x7f) << 14;
if (b < 0x80) {
return val;
}
- b = buf.get() & 0xff;
+ b = input.readByte() & 0xff;
val |= (b & 0x7f) << 21;
if (b < 0x80) {
return val;
}
val |= (b & 0x0f) << 28;
- return readVarIntRemainder(val, buf);
+ return readVarIntRemainder(input, val);
}
public static void rotate(long n, long[] xy, long rx, long ry) {
@@ -170,119 +174,119 @@ public class PMTiles {
private static final int HEADER_SIZE_BYTES = 127;
- public static Header decodeHeader(ByteBuffer buf) {
- buf.order(ByteOrder.LITTLE_ENDIAN);
+ public static Header deserializeHeader(LittleEndianDataInputStream input) throws IOException {
+ input.skipBytes(7);
return new Header(
- buf.get(7),
- buf.getLong(8),
- buf.getLong(16),
- buf.getLong(24),
- buf.getLong(32),
- buf.getLong(40),
- buf.getLong(48),
- buf.getLong(56),
- buf.getLong(64),
- buf.getLong(72),
- buf.getLong(80),
- buf.getLong(88),
- buf.get(96) == 1,
- Compression.values()[buf.get(97)],
- Compression.values()[buf.get(98)],
- TileType.values()[buf.get(99)],
- buf.get(100),
- buf.get(101),
- (double) buf.getInt(102) / 10000000,
- (double) buf.getInt(106) / 10000000,
- (double) buf.getInt(110) / 10000000,
- (double) buf.getInt(114) / 10000000,
- buf.get(118),
- (double) buf.getInt(119) / 10000000,
- (double) buf.getInt(123) / 10000000
- );
+ input.readByte(),
+ input.readLong(),
+ input.readLong(),
+ input.readLong(),
+ input.readLong(),
+ input.readLong(),
+ input.readLong(),
+ input.readLong(),
+ input.readLong(),
+ input.readLong(),
+ input.readLong(),
+ input.readLong(),
+ input.readByte() == 1,
+ Compression.values()[input.readByte()],
+ Compression.values()[input.readByte()],
+ TileType.values()[input.readByte()],
+ input.readByte(),
+ input.readByte(),
+ (double) input.readInt() / 10000000,
+ (double) input.readInt() / 10000000,
+ (double) input.readInt() / 10000000,
+ (double) input.readInt() / 10000000,
+ input.readByte(),
+ (double) input.readInt() / 10000000,
+ (double) input.readInt() / 10000000);
}
- public static void encodeHeader(Header header, ByteBuffer buf) {
- buf.order(ByteOrder.LITTLE_ENDIAN);
- buf.put(0, (byte) 0x50);
- buf.put(1, (byte) 0x4D);
- buf.put(2, (byte) 0x54);
- buf.put(3, (byte) 0x69);
- buf.put(4, (byte) 0x6C);
- buf.put(5, (byte) 0x65);
- buf.put(6, (byte) 0x73);
- buf.put(7, (byte) header.getSpecVersion());
- buf.putLong(8, header.getRootDirectoryOffset());
- buf.putLong(16, header.getRootDirectoryLength());
- buf.putLong(24, header.getJsonMetadataOffset());
- buf.putLong(32, header.getJsonMetadataLength());
- buf.putLong(40, header.getLeafDirectoryOffset());
- buf.putLong(48, header.getLeafDirectoryLength());
- buf.putLong(56, header.getTileDataOffset());
- buf.putLong(64, header.getTileDataLength());
- buf.putLong(72, header.getNumAddressedTiles());
- buf.putLong(80, header.getNumTileEntries());
- buf.putLong(88, header.getNumTileContents());
- buf.put(96, (byte) (header.isClustered() ? 1 : 0));
- buf.put(97, (byte) header.getInternalCompression().ordinal());
- buf.put(98, (byte) header.getTileCompression().ordinal());
- buf.put(99, (byte) header.getTileType().ordinal());
- buf.put(100, (byte) header.getMinZoom());
- buf.put(101, (byte) header.getMaxZoom());
- buf.putInt(102, (int) (header.getMinLon() * 10000000));
- buf.putInt(106, (int) (header.getMinLat() * 10000000));
- buf.putInt(110, (int) (header.getMaxLon() * 10000000));
- buf.putInt(114, (int) (header.getMaxLat() * 10000000));
- buf.put(118, (byte) header.getCenterZoom());
- buf.putInt(119, (int) (header.getCenterLon() * 10000000));
- buf.putInt(123, (int) (header.getCenterLat() * 10000000));
+ public static void serializeHeader(LittleEndianDataOutputStream output, Header header)
+ throws IOException {
+ output.writeByte((byte) 0x50);
+ output.writeByte((byte) 0x4D);
+ output.writeByte((byte) 0x54);
+ output.writeByte((byte) 0x69);
+ output.writeByte((byte) 0x6C);
+ output.writeByte((byte) 0x65);
+ output.writeByte((byte) 0x73);
+ output.writeByte((byte) header.getSpecVersion());
+ output.writeLong(header.getRootDirectoryOffset());
+ output.writeLong(header.getRootDirectoryLength());
+ output.writeLong(header.getJsonMetadataOffset());
+ output.writeLong(header.getJsonMetadataLength());
+ output.writeLong(header.getLeafDirectoryOffset());
+ output.writeLong(header.getLeafDirectoryLength());
+ output.writeLong(header.getTileDataOffset());
+ output.writeLong(header.getTileDataLength());
+ output.writeLong(header.getNumAddressedTiles());
+ output.writeLong(header.getNumTileEntries());
+ output.writeLong(header.getNumTileContents());
+ output.writeByte((byte) (header.isClustered() ? 1 : 0));
+ output.writeByte((byte) header.getInternalCompression().ordinal());
+ output.writeByte((byte) header.getTileCompression().ordinal());
+ output.writeByte((byte) header.getTileType().ordinal());
+ output.writeByte((byte) header.getMinZoom());
+ output.writeByte((byte) header.getMaxZoom());
+ output.writeInt((int) (header.getMinLon() * 10000000));
+ output.writeInt((int) (header.getMinLat() * 10000000));
+ output.writeInt((int) (header.getMaxLon() * 10000000));
+ output.writeInt((int) (header.getMaxLat() * 10000000));
+ output.writeByte((byte) header.getCenterZoom());
+ output.writeInt((int) (header.getCenterLon() * 10000000));
+ output.writeInt((int) (header.getCenterLat() * 10000000));
}
- public static void encodeDirectory(ByteBuffer buffer, List<Entry> entries) {
- buffer.order(ByteOrder.LITTLE_ENDIAN);
- encodeVarInt(buffer, entries.size());
+ public static void serializeEntries(LittleEndianDataOutputStream output, List<Entry> entries)
+ throws IOException {
+ writeVarInt(output, entries.size());
long lastId = 0;
for (Entry entry : entries) {
- encodeVarInt(buffer, entry.getTileId() - lastId);
+ writeVarInt(output, entry.getTileId() - lastId);
lastId = entry.getTileId();
}
for (Entry entry : entries) {
- encodeVarInt(buffer, entry.getRunLength());
+ writeVarInt(output, entry.getRunLength());
}
for (Entry entry : entries) {
- encodeVarInt(buffer, entry.getLength());
+ writeVarInt(output, entry.getLength());
}
- for (Entry entry : entries) {
- if (entry.getOffset() == 0 && entry.getLength() > 0) {
- Entry prevEntry = entries.get(entries.indexOf(entry) - 1);
- encodeVarInt(buffer, prevEntry.getOffset() + prevEntry.getLength() + 1);
+ for (int i = 0; i < entries.size(); i++) {
+ Entry entry = entries.get(i);
+ if (i > 0
+ && entry.getOffset() == entries.get(i - 1).getOffset() + entries.get(i - 1).getLength()) {
+ writeVarInt(output, 0);
} else {
- encodeVarInt(buffer, entry.getOffset() + 1);
+ writeVarInt(output, entry.getOffset() + 1);
}
}
}
- public static List<Entry> decodeDirectory(ByteBuffer buffer) {
- buffer.order(ByteOrder.LITTLE_ENDIAN);
- long numEntries = decodeVarInt(buffer);
+ public static List<Entry> deserializeEntries(LittleEndianDataInputStream buffer)
+ throws IOException {
+ long numEntries = readVarInt(buffer);
List<Entry> entries = new ArrayList<>((int) numEntries);
long lastId = 0;
for (int i = 0; i < numEntries; i++) {
- long value = decodeVarInt(buffer);
+ long value = readVarInt(buffer);
lastId = lastId + value;
Entry entry = new Entry();
entry.setTileId(lastId);
entries.add(entry);
}
for (int i = 0; i < numEntries; i++) {
- long value = decodeVarInt(buffer);
+ long value = readVarInt(buffer);
entries.get(i).setRunLength(value);
}
for (int i = 0; i < numEntries; i++) {
- long value = decodeVarInt(buffer);
+ long value = readVarInt(buffer);
entries.get(i).setLength(value);
}
for (int i = 0; i < numEntries; i++) {
- long value = decodeVarInt(buffer);
+ long value = readVarInt(buffer);
if (value == 0 && i > 0) {
Entry prevEntry = entries.get(i - 1);
entries.get(i).setOffset(prevEntry.getOffset() + prevEntry.getLength());;
@@ -320,4 +324,67 @@ public class PMTiles {
return null;
}
+ record Directories(byte[] root, byte[] leaves, int numLeaves) {
+ }
+
+ public static Directories buildRootLeaves(List<Entry> entries, int leafSize) throws IOException {
+ var rootEntries = new ArrayList<Entry>();
+ var numLeaves = 0;
+ byte[] leavesBytes = null;
+ byte[] rootBytes = null;
+
+ try (var leavesOutput = new ByteArrayOutputStream();
+ var leavesDataOutput = new LittleEndianDataOutputStream(leavesOutput)) {
+ for (var i = 0; i < entries.size(); i += leafSize) {
+ numLeaves++;
+ var end = i + leafSize;
+ if (i + leafSize > entries.size()) {
+ end = entries.size();
+ }
+
+ var offset = leavesOutput.size();
+ serializeEntries(leavesDataOutput, entries.subList(i, end));
+ var length = leavesOutput.size();
+ rootEntries.add(new Entry(entries.get(i).getTileId(), offset, length, 0));
+ }
+
+ leavesBytes = leavesOutput.toByteArray();
+ }
+
+ try (var rootOutput = new ByteArrayOutputStream();
+ var rootDataOutput = new LittleEndianDataOutputStream(rootOutput)) {
+ serializeEntries(rootDataOutput, rootEntries);
+ rootBytes = rootOutput.toByteArray();
+ }
+
+ return new Directories(rootBytes, leavesBytes, numLeaves);
+ }
+
+ public static Directories optimizeDirectories(List<Entry> entries, int targetRootLenght)
+ throws IOException {
+ if (entries.size() < 16384) {
+ byte[] rootBytes = null;
+ try (var rootOutput = new ByteArrayOutputStream();
+ var rootDataOutput = new LittleEndianDataOutputStream(rootOutput)) {
+ serializeEntries(rootDataOutput, entries);
+ rootBytes = rootOutput.toByteArray();
+ }
+ if (rootBytes.length <= targetRootLenght) {
+ return new Directories(rootBytes, new byte[] {}, 0);
+ }
+ }
+
+ double leafSize = (double) entries.size() / 3500d;
+ if (leafSize < 4096d) {
+ leafSize = 4096d;
+ }
+ for (;;) {
+ var directories = buildRootLeaves(entries, (int) leafSize);
+ if (directories.root.length <= targetRootLenght) {
+ return directories;
+ }
+ leafSize = leafSize * 1.2;
+ }
+
+ }
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java
new file mode 100644
index 00000000..ca0d45cf
--- /dev/null
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java
@@ -0,0 +1,91 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.baremaps.tilestore.pmtiles;
+
+import com.google.common.io.LittleEndianDataInputStream;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+
+public class PMTilesReader {
+
+ private final Path path;
+
+ private Header header;
+
+ private List<Entry> rootEntries;
+
+ public PMTilesReader(Path path) {
+ this.path = path;
+ }
+
+ public Header getHeader() {
+ if (header == null) {
+ try (var inputStream = new LittleEndianDataInputStream(Files.newInputStream(path))) {
+ header = PMTiles.deserializeHeader(inputStream);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ return header;
+ }
+
+ public List<Entry> getRootDirectory() {
+ if (rootEntries == null) {
+ var header = getHeader();
+ rootEntries =
+ getDirectory(header.getRootDirectoryOffset(), (int) header.getRootDirectoryLength());
+ }
+ return rootEntries;
+ }
+
+ public List<Entry> getDirectory(long offset, int length) {
+ var header = getHeader();
+ try (var input = Files.newInputStream(path)) {
+ input.skip(offset);
+ try (var decompressed =
+ new LittleEndianDataInputStream(header.getInternalCompression().decompress(input))) {
+ return PMTiles.deserializeEntries(decompressed);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public ByteBuffer getTile(int z, long x, long y) {
+ var tileId = PMTiles.zxyToTileId(z, x, y);
+ var header = getHeader();
+ var entries = getRootDirectory();
+ var entry = PMTiles.findTile(entries, tileId);
+
+ if (entry == null) {
+ return null;
+ }
+
+ try (var channel = FileChannel.open(path)) {
+ var compressed = ByteBuffer.allocate((int) entry.getLength());
+ channel.position(header.getTileDataOffset() + entry.getOffset());
+ channel.read(compressed);
+ compressed.flip();
+ try (var tile = new ByteArrayInputStream(compressed.array())) {
+ return ByteBuffer.wrap(tile.readAllBytes());
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/file/FileTileStore.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesStore.java
similarity index 59%
copy from baremaps-core/src/main/java/org/apache/baremaps/tilestore/file/FileTileStore.java
copy to baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesStore.java
index 77424004..5047e333 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/file/FileTileStore.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesStore.java
@@ -10,66 +10,52 @@
* the License.
*/
-package org.apache.baremaps.tilestore.file;
-
-
+package org.apache.baremaps.tilestore.pmtiles;
import java.io.IOException;
import java.nio.ByteBuffer;
-import java.nio.file.Files;
import java.nio.file.Path;
import org.apache.baremaps.tilestore.TileCoord;
import org.apache.baremaps.tilestore.TileStore;
import org.apache.baremaps.tilestore.TileStoreException;
-/** Represents a {@code TileStore} baked by a directory. */
-public class FileTileStore implements TileStore {
+public class PMTilesStore implements TileStore {
- private final Path path;
+ private final PMTilesWriter writer;
- /**
- * Constructs a {@code FileTileStore}.
- *
- * @param path the directory
- */
- public FileTileStore(Path path) {
- this.path = path;
+ public PMTilesStore(Path path) {
+ try {
+ this.writer = new PMTilesWriter(path);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
}
- /** {@inheritDoc} */
@Override
public ByteBuffer read(TileCoord tileCoord) throws TileStoreException {
- try {
- return ByteBuffer.wrap(Files.readAllBytes(resolve(tileCoord)));
- } catch (IOException e) {
- throw new TileStoreException(e);
- }
+ throw new UnsupportedOperationException();
}
- /** {@inheritDoc} */
@Override
public void write(TileCoord tileCoord, ByteBuffer blob) throws TileStoreException {
try {
- var file = resolve(tileCoord);
- Files.createDirectories(file.getParent());
- Files.write(file, blob.array());
+ writer.writeTile(tileCoord.z(), tileCoord.x(), tileCoord.y(), blob.array());
} catch (IOException e) {
throw new TileStoreException(e);
}
}
- /** {@inheritDoc} */
@Override
public void delete(TileCoord tileCoord) throws TileStoreException {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public void close() throws TileStoreException {
try {
- Files.deleteIfExists(resolve(tileCoord));
+ writer.finalize();
} catch (IOException e) {
throw new TileStoreException(e);
}
}
-
- /** {@inheritDoc} */
- public Path resolve(TileCoord tileCoord) {
- return path.resolve(String.format("%s/%s/%s.mvt", tileCoord.z(), tileCoord.x(), tileCoord.y()));
- }
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesWriter.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesWriter.java
new file mode 100644
index 00000000..267f6047
--- /dev/null
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTilesWriter.java
@@ -0,0 +1,119 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.baremaps.tilestore.pmtiles;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.io.LittleEndianDataOutputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class PMTilesWriter {
+
+ private final Path path;
+
+ private List<Entry> entries;
+
+ private Path tilePath;
+
+ private boolean clustered = true;
+
+ public PMTilesWriter(Path path) throws IOException {
+ this.path = path;
+ this.entries = new ArrayList<>();
+ this.tilePath = Files.createTempFile(path.getParent(), "tiles", ".tmp");
+ }
+
+ public void writeTile(int z, int x, int y, byte[] bytes) throws IOException {
+ var tileId = PMTiles.zxyToTileId(z, x, y);
+ var offset = Files.size(tilePath);
+ var length = bytes.length;
+
+ if (entries.size() > 0 && tileId < entries.get(entries.size() - 1).getTileId()) {
+ clustered = false;
+ }
+
+ try (var output = new FileOutputStream(tilePath.toFile(), true)) {
+ output.write(bytes);
+ entries.add(new Entry(tileId, offset, length, 1));
+ }
+ }
+
+ public void finalize() throws IOException {
+ // Sort the entries by tile id
+ if (!clustered) {
+ entries.sort(Comparator.comparingLong(Entry::getTileId));
+ }
+
+ var metadataMap = new HashMap<String, Object>();
+ metadataMap.put("name", "baremaps");
+ metadataMap.put("type", "baselayer");
+ metadataMap.put("version", "0.0.1");
+ metadataMap.put("description", "PMTiles generated by Baremaps");
+ metadataMap.put("attribution", "OpenStreetMap contributors");
+ metadataMap.put("vector_layers", List.of("aerialway", "aeroway", "amenity", "attraction",
+ "barrier", "boundary", "building", "highway", "landuse",
+ "leisure", "man_made", "natural", "ocean", "point",
+ "power", "railway", "route", "waterway")
+ .stream().map(s -> Map.of("id", s)).toList());
+
+ var metadata = new ObjectMapper().writeValueAsBytes(metadataMap);
+
+
+ var directories = PMTiles.optimizeDirectories(entries, 16247);
+ long rootOffset = 127;
+ long rootLength = directories.root().length;
+ long metadataOffset = rootOffset + rootLength;
+ long metadataLength = metadata.length;
+ long leavesOffset = metadataOffset + metadataLength;
+ long leavesLength = directories.leaves().length;
+ long tilesOffset = leavesOffset + leavesLength;
+ long tilesLength = Files.size(tilePath);
+ int minZoom = (int) PMTiles.tileIdToZxy(entries.get(0).getTileId())[0];
+ int maxZoom = (int) PMTiles.tileIdToZxy(entries.get(entries.size() - 1).getTileId())[0];
+ var numTiles = entries.size();
+
+ var header = new Header();
+ header.setNumAddressedTiles(numTiles);
+ header.setNumTileEntries(numTiles);
+ header.setNumTileContents(numTiles);
+ header.setClustered(true);
+ header.setInternalCompression(Compression.None);
+ header.setTileCompression(Compression.Gzip);
+ header.setTileType(TileType.Mvt);
+ header.setMinZoom(minZoom);
+ header.setMaxZoom(maxZoom);
+ header.setRootOffset(rootOffset);
+ header.setRootLength(rootLength);
+ header.setMetadataOffset(metadataOffset);
+ header.setMetadataLength(metadataLength);
+ header.setLeavesOffset(leavesOffset);
+ header.setLeavesLength(leavesLength);
+ header.setTilesOffset(tilesOffset);
+ header.setTilesLength(tilesLength);
+
+
+ try (var output = new LittleEndianDataOutputStream(new FileOutputStream(path.toFile()))) {
+ PMTiles.serializeHeader(output, header);
+ output.write(directories.root());
+ output.write(metadata);
+ output.write(directories.leaves());
+ Files.copy(tilePath, output);
+ } finally {
+ Files.delete(tilePath);
+ }
+ }
+
+}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/TileType.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/TileType.java
index 10bf41c8..f52673c1 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/TileType.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/TileType.java
@@ -1,10 +1,22 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
+ * in compliance with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
package org.apache.baremaps.tilestore.pmtiles;
enum TileType {
- Unknown,
- Mvt,
- Png,
- Jpeg,
- Webp,
- Avif,
+ Unknown,
+ Mvt,
+ Png,
+ Jpeg,
+ Webp,
+ Avif,
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/postgres/PostgresTileStore.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/postgres/PostgresTileStore.java
index 7b4f75fb..039f6bd2 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/postgres/PostgresTileStore.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/postgres/PostgresTileStore.java
@@ -258,4 +258,8 @@ public class PostgresTileStore implements TileStore {
throw new UnsupportedOperationException("The postgis tile store is read only");
}
+ @Override
+ public void close() throws Exception {
+ // do nothing
+ }
}
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/workflow/tasks/ExportVectorTiles.java b/baremaps-core/src/main/java/org/apache/baremaps/workflow/tasks/ExportVectorTiles.java
index 5b9f672e..d900611d 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/workflow/tasks/ExportVectorTiles.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/workflow/tasks/ExportVectorTiles.java
@@ -29,6 +29,7 @@ import org.apache.baremaps.stream.StreamUtils;
import org.apache.baremaps.tilestore.*;
import org.apache.baremaps.tilestore.file.FileTileStore;
import org.apache.baremaps.tilestore.mbtiles.MBTilesStore;
+import org.apache.baremaps.tilestore.pmtiles.PMTilesStore;
import org.apache.baremaps.tilestore.postgres.PostgresTileStore;
import org.apache.baremaps.utils.SqliteUtils;
import org.apache.baremaps.vectortile.tileset.Tileset;
@@ -48,7 +49,8 @@ public record ExportVectorTiles(
public enum Format {
file,
- mbtiles
+ mbtiles,
+ pmtiles
}
private static final Logger logger = LoggerFactory.getLogger(ExportVectorTiles.class);
@@ -60,22 +62,24 @@ public record ExportVectorTiles(
var tileset = objectMapper.readValue(configReader.read(this.tileset), Tileset.class);
var datasource = context.getDataSource(tileset.getDatabase());
- var sourceTileStore = sourceTileStore(tileset, datasource);
- var targetTileStore = targetTileStore(tileset);
+ try (var sourceTileStore = sourceTileStore(tileset, datasource);
+ var targetTileStore = targetTileStore(tileset)) {
- var envelope = tileset.getBounds().size() == 4
- ? new Envelope(
- tileset.getBounds().get(0), tileset.getBounds().get(2),
- tileset.getBounds().get(1), tileset.getBounds().get(3))
- : new Envelope(-180, 180, -85.0511, 85.0511);
+ var envelope = tileset.getBounds().size() == 4
+ ? new Envelope(
+ tileset.getBounds().get(0), tileset.getBounds().get(2),
+ tileset.getBounds().get(1), tileset.getBounds().get(3))
+ : new Envelope(-180, 180, -85.0511, 85.0511);
- var count = TileCoord.count(envelope, tileset.getMinzoom(), tileset.getMaxzoom());
+ var count = TileCoord.count(envelope, tileset.getMinzoom(), tileset.getMaxzoom());
- var stream =
- StreamUtils.stream(TileCoord.iterator(envelope, tileset.getMinzoom(), tileset.getMaxzoom()))
- .peek(new ProgressLogger<>(count, 5000));
+ var stream =
+ StreamUtils
+ .stream(TileCoord.iterator(envelope, tileset.getMinzoom(), tileset.getMaxzoom()))
+ .peek(new ProgressLogger<>(count, 5000));
- StreamUtils.batch(stream).forEach(new TileChannel(sourceTileStore, targetTileStore));
+ StreamUtils.batch(stream).forEach(new TileChannel(sourceTileStore, targetTileStore));
+ }
}
private TileStore sourceTileStore(Tileset tileset, DataSource datasource) {
@@ -93,6 +97,10 @@ public record ExportVectorTiles(
tilesStore.initializeDatabase();
tilesStore.writeMetadata(metadata(source));
return tilesStore;
+ case pmtiles:
+ Files.deleteIfExists(repository);
+ var tileStore = new PMTilesStore(repository);
+ return tileStore;
default:
throw new IllegalArgumentException("Unsupported format");
}
diff --git a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java
deleted file mode 100644
index 4ca92112..00000000
--- a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesReader.java
+++ /dev/null
@@ -1,77 +0,0 @@
-package org.apache.baremaps.tilestore.pmtiles;
-
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.List;
-
-public class PMTilesReader {
-
- private final Path path;
-
- private Header header;
-
- private List<Entry> rootEntries;
-
- public PMTilesReader(Path path) {
- this.path = path;
- }
-
- public Header getHeader() {
- if (header == null) {
- try (var channel = Files.newByteChannel(path)) {
- var buffer = ByteBuffer.allocate(127);
- channel.read(buffer);
- buffer.flip();
- header = PMTiles.decodeHeader(buffer);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
- return header;
- }
-
- public List<Entry> getRootDirectory() {
- if (rootEntries == null) {
- var header = getHeader();
- rootEntries = getDirectory(header.getRootDirectoryOffset(), (int) header.getRootDirectoryLength());
- }
- return rootEntries;
- }
-
- public List<Entry> getDirectory(long offset, int length) {
- var header = getHeader();
- try (var channel = Files.newByteChannel(path)) {
- var compressed = ByteBuffer.allocate(length);
- channel.position(offset);
- channel.read(compressed);
- compressed.flip();
- var decompressed = header.getInternalCompression().decompress(compressed);
- return PMTiles.decodeDirectory(decompressed);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- public ByteBuffer getTile(int z, long x, long y) {
- var tileId = PMTiles.zxyToTileId(z, x, y);
- var header = getHeader();
- var entries = getRootDirectory();
- var entry = PMTiles.findTile(entries, tileId);
-
- if (entry == null) {
- return null;
- }
-
- try (var channel = Files.newByteChannel(path)) {
- var compressed = ByteBuffer.allocate((int) entry.getLength());
- channel.position(header.getTileDataOffset() + entry.getOffset());
- channel.read(compressed);
- compressed.flip();
- return header.getTileCompression().decompress(compressed);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-}
diff --git a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
index 84211805..fbe7fcd7 100644
--- a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
+++ b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
@@ -14,57 +14,57 @@ package org.apache.baremaps.tilestore.pmtiles;
import static org.junit.jupiter.api.Assertions.*;
+import com.google.common.io.LittleEndianDataInputStream;
+import com.google.common.io.LittleEndianDataOutputStream;
import com.google.common.math.LongMath;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.file.Files;
+import java.nio.channels.Channels;
+import java.nio.channels.FileChannel;
import java.util.ArrayList;
-
-import com.google.protobuf.InvalidProtocolBufferException;
+import java.util.List;
+import java.util.Random;
import org.apache.baremaps.testing.TestFiles;
-import org.apache.baremaps.vectortile.VectorTileDecoder;
-import org.apache.baremaps.vectortile.VectorTileViewer;
-import org.apache.baremaps.vectortile.VectorTileViewer.TilePanel;
import org.junit.jupiter.api.Test;
-import javax.swing.*;
-
class PMTilesTest {
@Test
- void decodeVarInt() {
- var b = ByteBuffer.wrap(new byte[] {
+ void decodeVarInt() throws IOException {
+ var b = new LittleEndianDataInputStream(new ByteArrayInputStream(new byte[] {
(byte) 0, (byte) 1,
(byte) 127, (byte) 0xe5,
(byte) 0x8e, (byte) 0x26
- });
- assertEquals(PMTiles.decodeVarInt(b), 0);
- assertEquals(PMTiles.decodeVarInt(b), 1);
- assertEquals(PMTiles.decodeVarInt(b), 127);
- assertEquals(PMTiles.decodeVarInt(b), 624485);
- b = ByteBuffer.wrap(new byte[] {
+ }));
+ assertEquals(PMTiles.readVarInt(b), 0);
+ assertEquals(PMTiles.readVarInt(b), 1);
+ assertEquals(PMTiles.readVarInt(b), 127);
+ assertEquals(PMTiles.readVarInt(b), 624485);
+ b = new LittleEndianDataInputStream(new ByteArrayInputStream(new byte[] {
(byte) 0xff, (byte) 0xff,
(byte) 0xff, (byte) 0xff,
(byte) 0xff, (byte) 0xff,
(byte) 0xff, (byte) 0x0f,
- });
- assertEquals(PMTiles.decodeVarInt(b), 9007199254740991L);
+ }));
+ assertEquals(PMTiles.readVarInt(b), 9007199254740991L);
}
@Test
- void encodeVarInt() {
- var buffer = ByteBuffer.allocate(10);
+ void encodeVarInt() throws IOException {
for (long i = 0; i < 1000; i++) {
- PMTiles.encodeVarInt(buffer, i);
- buffer.flip();
- assertEquals(i, PMTiles.decodeVarInt(buffer));
- buffer.clear();
+ var array = new ByteArrayOutputStream();
+ var output = new LittleEndianDataOutputStream(array);
+ PMTiles.writeVarInt(output, i);
+ var input = new LittleEndianDataInputStream(new ByteArrayInputStream(array.toByteArray()));
+ assertEquals(i, PMTiles.readVarInt(input));
}
for (long i = Long.MAX_VALUE - 1000; i < Long.MAX_VALUE; i++) {
- PMTiles.encodeVarInt(buffer, i);
- buffer.flip();
- assertEquals(i, PMTiles.decodeVarInt(buffer));
- buffer.clear();
+ var array = new ByteArrayOutputStream();
+ var output = new LittleEndianDataOutputStream(array);
+ PMTiles.writeVarInt(output, i);
+ var input = new LittleEndianDataInputStream(new ByteArrayInputStream(array.toByteArray()));
+ assertEquals(i, PMTiles.readVarInt(input));
}
}
@@ -127,11 +127,9 @@ class PMTilesTest {
@Test
void decodeHeader() throws IOException {
var file = TestFiles.resolve("pmtiles/test_fixture_1.pmtiles");
- try (var channel = Files.newByteChannel(file)) {
- var buffer = ByteBuffer.allocate(127);
- channel.read(buffer);
- buffer.flip();
- var header = PMTiles.decodeHeader(buffer);
+ try (var channel = FileChannel.open(file)) {
+ var input = new LittleEndianDataInputStream(Channels.newInputStream(channel));
+ var header = PMTiles.deserializeHeader(input);
assertEquals(header.getRootDirectoryOffset(), 127);
assertEquals(header.getRootDirectoryLength(), 25);
assertEquals(header.getJsonMetadataOffset(), 152);
@@ -159,7 +157,6 @@ class PMTilesTest {
@Test
void encodeHeader() throws IOException {
var etag = "1";
- var buffer = ByteBuffer.allocate(127);
var header = new Header(
127,
25,
@@ -185,10 +182,18 @@ class PMTilesTest {
0,
0,
0,
- 0
- );
- PMTiles.encodeHeader(header, buffer);
- var header2 = PMTiles.decodeHeader(buffer);
+ 0);
+
+ var array = new ByteArrayOutputStream();
+
+ var output = new LittleEndianDataOutputStream(array);
+ PMTiles.serializeHeader(output, header);
+
+ var array2 = array.toByteArray();
+
+ var input = new LittleEndianDataInputStream(new ByteArrayInputStream(array.toByteArray()));
+ var header2 = PMTiles.deserializeHeader(input);
+
assertEquals(header, header2);
}
@@ -249,26 +254,32 @@ class PMTilesTest {
}
@Test
- void reader() throws InvalidProtocolBufferException, InterruptedException {
- var reader = new PMTilesReader(TestFiles.resolve("pmtiles/test_fixture_1.pmtiles"));
- var header = reader.getHeader();
- assertEquals(header.getRootDirectoryOffset(), 127);
- var rootDirectory = reader.getRootDirectory();
- assertEquals(rootDirectory.size(), 1);
- var entry = rootDirectory.get(0);
- ByteBuffer buffer = reader.getTile(0,0,0);
+ void buildRootLeaves() throws IOException {
+ var entries = List.of(new Entry(100, 1, 1, 0));
+ var directories = PMTiles.buildRootLeaves(entries, 1);
+ assertEquals(directories.numLeaves(), 1);
- VectorTileViewer viewer = new VectorTileViewer();
- var parsed = org.apache.baremaps.mvt.binary.VectorTile.Tile.parseFrom(buffer);
- var tile = new VectorTileDecoder().decodeTile(parsed);
- JFrame f = new JFrame("Vector Tile Viewer");
- f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
- f.add(new TilePanel(tile, 1000));
- f.pack();
- f.setVisible(true);
-
- Thread.sleep(10000);
}
+ @Test
+ void optimizeDirectories() throws IOException {
+ var random = new Random(3857);
+ var entries = new ArrayList<Entry>();
+ entries.add(new Entry(0, 0, 100, 1));
+ var directories = PMTiles.optimizeDirectories(entries, 100);
+ assertFalse(directories.leaves().length > 0);
+ assertEquals(0, directories.numLeaves());
+ entries = new ArrayList<>();
+ int offset = 0;
+ for (var i = 0; i < 1000; i++) {
+ var randTileSize = random.nextInt(1000000);
+ entries.add(new Entry(i, offset, randTileSize, 1));
+ offset += randTileSize;
+ }
+ directories = PMTiles.optimizeDirectories(entries, 1024);
+ assertFalse(directories.root().length > 1024);
+ assertFalse(directories.numLeaves() == 0);
+ assertFalse(directories.leaves().length == 0);
+ }
}
[incubator-baremaps] 04/08: Port additional methods and tests
Posted by bc...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bchapuis pushed a commit to branch pmtiles
in repository https://gitbox.apache.org/repos/asf/incubator-baremaps.git
commit 5ef6a50f519ca9a8b3d266c6d375509acf2d980a
Author: Bertil Chapuis <bc...@gmail.com>
AuthorDate: Thu Sep 7 00:11:57 2023 +0200
Port additional methods and tests
---
.../apache/baremaps/tilestore/pmtiles/PMTiles.java | 158 ++++++++++++++++++++-
.../baremaps/tilestore/pmtiles/PMTilesTest.java | 99 +++++++++++--
2 files changed, 241 insertions(+), 16 deletions(-)
diff --git a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
index 3ec6d62e..1ff45353 100644
--- a/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
+++ b/baremaps-core/src/main/java/org/apache/baremaps/tilestore/pmtiles/PMTiles.java
@@ -15,6 +15,8 @@ package org.apache.baremaps.tilestore.pmtiles;
import com.google.common.math.LongMath;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.List;
public class PMTiles {
@@ -22,7 +24,7 @@ public class PMTiles {
return high * 0x100000000L + low;
}
- public static long readVarintRemainder(long l, ByteBuffer buf) {
+ public static long readVarIntRemainder(long l, ByteBuffer buf) {
long h, b;
b = buf.get() & 0xff;
h = (b & 0x70) >> 4;
@@ -57,7 +59,18 @@ public class PMTiles {
throw new RuntimeException("Expected varint not more than 10 bytes");
}
- public static long readVarint(ByteBuffer buf) {
+ public static int encodeVarInt(ByteBuffer buf, long value) {
+ int n = 1;
+ while (value >= 0x80) {
+ buf.put((byte) (value | 0x80));
+ value >>>= 7;
+ n++;
+ }
+ buf.put((byte) value);
+ return n;
+ }
+
+ public static long decodeVarInt(ByteBuffer buf) {
long val, b;
b = buf.get() & 0xff;
val = b & 0x7f;
@@ -80,7 +93,7 @@ public class PMTiles {
return val;
}
val |= (b & 0x0f) << 28;
- return readVarintRemainder(val, buf);
+ return readVarIntRemainder(val, buf);
}
public static void rotate(long n, long[] xy, long rx, long ry) {
@@ -172,6 +185,8 @@ public class PMTiles {
Avif,
}
+ private static final int HEADER_SIZE_BYTES = 127;
+
public record Header(
int specVersion,
long rootDirectoryOffset,
@@ -201,7 +216,7 @@ public class PMTiles {
String etag) {
}
- public static Header bytesToHeader(ByteBuffer buf, String etag) {
+ public static Header decodeHeader(ByteBuffer buf, String etag) {
buf.order(ByteOrder.LITTLE_ENDIAN);
return new Header(
buf.get(7),
@@ -232,7 +247,7 @@ public class PMTiles {
etag);
}
- public static void headerToBytes(Header header, ByteBuffer buf) {
+ public static void encodeHeader(Header header, ByteBuffer buf) {
buf.order(ByteOrder.LITTLE_ENDIAN);
buf.put(0, (byte) 0x50);
buf.put(1, (byte) 0x4D);
@@ -268,4 +283,137 @@ public class PMTiles {
buf.putInt(123, (int) (header.centerLat * 10000000));
}
+ public static class Entry {
+ private long tileId;
+ private long offset;
+ private long length;
+ private long runLength;
+
+ public Entry() {
+
+ }
+
+ public Entry(long tileId, long offset, long length, long runLength) {
+ this.tileId = tileId;
+ this.offset = offset;
+ this.length = length;
+ this.runLength = runLength;
+ }
+
+ public long getTileId() {
+ return tileId;
+ }
+
+ public void setTileId(long tileId) {
+ this.tileId = tileId;
+ }
+
+ public long getOffset() {
+ return offset;
+ }
+
+ public void setOffset(long offset) {
+ this.offset = offset;
+ }
+
+ public long getLength() {
+ return length;
+ }
+
+ public void setLength(long length) {
+ this.length = length;
+ }
+
+ public long getRunLength() {
+ return runLength;
+ }
+
+ public void setRunLength(long runLength) {
+ this.runLength = runLength;
+ }
+ }
+
+ public static void encodeDirectory(ByteBuffer buffer, List<Entry> entries) {
+ buffer.order(ByteOrder.LITTLE_ENDIAN);
+ encodeVarInt(buffer, entries.size());
+ long lastId = 0;
+ for (Entry entry : entries) {
+ encodeVarInt(buffer, entry.getTileId() - lastId);
+ lastId = entry.getTileId();
+ }
+ for (Entry entry : entries) {
+ encodeVarInt(buffer, entry.getRunLength());
+ }
+ for (Entry entry : entries) {
+ encodeVarInt(buffer, entry.getLength());
+ }
+ for (Entry entry : entries) {
+ if (entry.getOffset() == 0 && entry.getLength() > 0) {
+ Entry prevEntry = entries.get(entries.indexOf(entry) - 1);
+ encodeVarInt(buffer, prevEntry.offset + prevEntry.length + 1);
+ } else {
+ encodeVarInt(buffer, entry.getOffset() + 1);
+ }
+ }
+ }
+
+ public static List<Entry> decodeDirectory(ByteBuffer buffer) {
+ buffer.order(ByteOrder.LITTLE_ENDIAN);
+ long numEntries = decodeVarInt(buffer);
+ List<Entry> entries = new ArrayList<>((int) numEntries);
+ long lastId = 0;
+ for (int i = 0; i < numEntries; i++) {
+ long value = decodeVarInt(buffer);
+ lastId = lastId + value;
+ Entry entry = new Entry();
+ entry.setTileId(lastId);
+ entries.add(entry);
+ }
+ for (int i = 0; i < numEntries; i++) {
+ long value = decodeVarInt(buffer);
+ entries.get(i).setRunLength(value);
+ }
+ for (int i = 0; i < numEntries; i++) {
+ long value = decodeVarInt(buffer);
+ entries.get(i).setLength(value);
+ }
+ for (int i = 0; i < numEntries; i++) {
+ long value = decodeVarInt(buffer);
+ if (value == 0 && i > 0) {
+ Entry prevEntry = entries.get(i - 1);
+ entries.get(i).setOffset(prevEntry.offset + prevEntry.length);;
+ } else {
+ entries.get(i).setOffset(value - 1);
+ }
+ }
+ return entries;
+ }
+
+ public static Entry findTile(List<Entry> entries, long tileId) {
+ int m = 0;
+ int n = entries.size() - 1;
+ while (m <= n) {
+ int k = (n + m) >> 1;
+ long cmp = tileId - entries.get(k).getTileId();
+ if (cmp > 0) {
+ m = k + 1;
+ } else if (cmp < 0) {
+ n = k - 1;
+ } else {
+ return entries.get(k);
+ }
+ }
+
+ // at this point, m > n
+ if (n >= 0) {
+ if (entries.get(n).runLength == 0) {
+ return entries.get(n);
+ }
+ if (tileId - entries.get(n).tileId < entries.get(n).runLength) {
+ return entries.get(n);
+ }
+ }
+ return null;
+ }
+
}
diff --git a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
index 1fdf7129..b5bea852 100644
--- a/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
+++ b/baremaps-core/src/test/java/org/apache/baremaps/tilestore/pmtiles/PMTilesTest.java
@@ -18,31 +18,51 @@ import com.google.common.math.LongMath;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Optional;
import org.apache.baremaps.testing.TestFiles;
import org.apache.baremaps.tilestore.pmtiles.PMTiles.Compression;
+import org.apache.baremaps.tilestore.pmtiles.PMTiles.Entry;
import org.apache.baremaps.tilestore.pmtiles.PMTiles.TileType;
import org.junit.jupiter.api.Test;
class PMTilesTest {
@Test
- void varint() {
+ void decodeVarInt() {
var b = ByteBuffer.wrap(new byte[] {
(byte) 0, (byte) 1,
(byte) 127, (byte) 0xe5,
(byte) 0x8e, (byte) 0x26
});
- assertEquals(PMTiles.readVarint(b), 0);
- assertEquals(PMTiles.readVarint(b), 1);
- assertEquals(PMTiles.readVarint(b), 127);
- assertEquals(PMTiles.readVarint(b), 624485);
+ assertEquals(PMTiles.decodeVarInt(b), 0);
+ assertEquals(PMTiles.decodeVarInt(b), 1);
+ assertEquals(PMTiles.decodeVarInt(b), 127);
+ assertEquals(PMTiles.decodeVarInt(b), 624485);
b = ByteBuffer.wrap(new byte[] {
(byte) 0xff, (byte) 0xff,
(byte) 0xff, (byte) 0xff,
(byte) 0xff, (byte) 0xff,
(byte) 0xff, (byte) 0x0f,
});
- assertEquals(PMTiles.readVarint(b), 9007199254740991L);
+ assertEquals(PMTiles.decodeVarInt(b), 9007199254740991L);
+ }
+
+ @Test
+ void encodeVarInt() {
+ var buffer = ByteBuffer.allocate(10);
+ for (long i = 0; i < 1000; i++) {
+ PMTiles.encodeVarInt(buffer, i);
+ buffer.flip();
+ assertEquals(i, PMTiles.decodeVarInt(buffer));
+ buffer.clear();
+ }
+ for (long i = Long.MAX_VALUE - 1000; i < Long.MAX_VALUE; i++) {
+ PMTiles.encodeVarInt(buffer, i);
+ buffer.flip();
+ assertEquals(i, PMTiles.decodeVarInt(buffer));
+ buffer.clear();
+ }
}
@Test
@@ -102,13 +122,13 @@ class PMTilesTest {
}
@Test
- void bytesToHeader() throws IOException {
+ void decodeHeader() throws IOException {
var file = TestFiles.resolve("pmtiles/test_fixture_1.pmtiles");
try (var channel = Files.newByteChannel(file)) {
var buffer = ByteBuffer.allocate(127);
channel.read(buffer);
buffer.flip();
- var header = PMTiles.bytesToHeader(buffer, "1");
+ var header = PMTiles.decodeHeader(buffer, "1");
assertEquals(header.rootDirectoryOffset(), 127);
assertEquals(header.rootDirectoryLength(), 25);
assertEquals(header.jsonMetadataOffset(), 152);
@@ -134,7 +154,7 @@ class PMTilesTest {
}
@Test
- void headerToBytes() throws IOException {
+ void encodeHeader() throws IOException {
var etag = "1";
var buffer = ByteBuffer.allocate(127);
var header = new PMTiles.Header(
@@ -164,9 +184,66 @@ class PMTilesTest {
0,
0,
etag);
- PMTiles.headerToBytes(header, buffer);
- var header2 = PMTiles.bytesToHeader(buffer, etag);
+ PMTiles.encodeHeader(header, buffer);
+ var header2 = PMTiles.decodeHeader(buffer, etag);
assertEquals(header, header2);
}
+ @Test
+ void searchForMissingEntry() {
+ var entries = new ArrayList<Entry>();
+ assertEquals(PMTiles.findTile(entries, 101), Optional.empty());
+ }
+
+ @Test
+ void searchForFirstEntry() {
+ var entry = new Entry(100, 1, 1, 1);
+ var entries = new ArrayList<Entry>();
+ entries.add(entry);
+ assertEquals(PMTiles.findTile(entries, 100), Optional.of(entry));
+ }
+
+ @Test
+ void searchWithRunLength() {
+ var entry = new Entry(3, 3, 1, 2);
+ var entries = new ArrayList<Entry>();
+ entries.add(entry);
+ entries.add(new Entry(5, 5, 1, 2));
+ assertEquals(PMTiles.findTile(entries, 4), Optional.of(entry));
+ }
+
+ @Test
+ void searchWithMultipleTileEntries() {
+ var entries = new ArrayList<Entry>();
+ entries.add(new Entry(100, 1, 1, 2));
+ var entry = PMTiles.findTile(entries, 101);
+ assertEquals(entry.getOffset(), 1);
+ assertEquals(entry.getLength(), 1);
+
+ entries = new ArrayList<Entry>();
+ entries.add(new Entry(100, 1, 1, 1));
+ entries.add(new Entry(150, 2, 2, 2));
+ entry = PMTiles.findTile(entries, 151);
+ assertEquals(entry.getOffset(), 2);
+ assertEquals(entry.getLength(), 2);
+
+ entries = new ArrayList<>();
+ entries.add(new Entry(50, 1, 1, 2));
+ entries.add(new Entry(100, 2, 2, 1));
+ entries.add(new Entry(150, 3, 3, 1));
+ entry = PMTiles.findTile(entries, 51);
+ assertEquals(entry.getOffset(), 1);
+ assertEquals(entry.getLength(), 1);
+ }
+
+ @Test
+ void leafSearch() {
+ var entries = new ArrayList<Entry>();
+ entries.add(new Entry(100, 1, 1, 0));
+ var entry = PMTiles.findTile(entries, 150);
+ assertEquals(entry.getOffset(), 1);
+ assertEquals(entry.getLength(), 1);
+ }
+
+
}