[GitHub] [incubator-baremaps] bchapuis commented on issue #574: Update OSM documentation

["bchapuis (via GitHub)" <gi...@apache.org>]

bchapuis commented on issue #574:
URL: https://github.com/apache/incubator-baremaps/issues/574#issuecomment-1407699850

   
   > * importer: Is the importer custom to this package, or using an existing library?  What is the performance of that import process for the world pbf, compared to `Osm2pgsql`, `Osmium`, `Osmosis` and `Imposm`.
   
   The pbf parser and database importer are custom to this project. The parser use the Java Stream api to deserialise the pbf in parallel. The importer uses the postgres COPY api to achieve great performance. They should be in par with what other projects are doing.
   
   Our current performance bottleneck is related to geometry simplification (merging highways, creating simplified polygons for landuse, etc.). This currently occurs once the data has been imported in the database. For the planet, this part of the process takes about two days and we are currently trying to optimise [this part of the workflow](https://github.com/apache/incubator-baremaps/pull/570).
   
   > * data structure:  It sounds like this is a full import?  That it maintains all  nodes, ways and relations?
   
   Yes, we import all node, ways, relations and tags. 
   
   > * normalization: `In order to save denormalized geometries in PostGIS (e.g. linestring, polygon, multi-polygon, etc.), Baremaps creates a cache for nodes, ways and relations. [LMDB](https://symas.com/lmdb/) is used under the hood to achieve great performance.`  It sounds like this normalization is happening in the server and not in postgres.  What is the performance impact of this normalization? Is there a `cache warming` period on server restart?
   
   This part of the documentation is outdated. We removed LMDB and now rely on memory mapped files to cache the nodes and references. As the pbf files are ordered, we are creating these caches on the fly and read the pbf files only once.
   
   > * updating:  Any plans on supporting periodic updates from OSM changesets from something like `osmupdate`?
   
   Yes, we do have preliminary support for [osm updates](https://github.com/apache/incubator-baremaps/blob/main/baremaps-core/src/main/java/org/apache/baremaps/workflow/tasks/UpdateOpenStreetMap.java) but it is experimental and not extensively tested.
   
   > * caching: It appears to me like the server is caching tiles. It would be helpful to know how that caching is happening, what are the space requirements, and how one would purge tiles from cache (especially after a changeset has been imported).
   
   Yes, the server uses [Caffeine](https://github.com/ben-manes/caffeine) to cache tiles in the RAM
   
   The configuration of the cache can be tweaked with the [`--cache` flag](https://github.com/apache/incubator-baremaps/blob/48b1ce7a85a1701c37433a3bbac5044244261788/baremaps-cli/src/main/java/org/apache/baremaps/cli/map/Serve.java#LL53
   ) of the map serve command. It  takes a [caffeine specification](https://www.javadoc.io/doc/com.github.ben-manes.caffeine/caffeine/2.2.2/com/github/benmanes/caffeine/cache/CaffeineSpec.html) as an argument.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org