You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@baremaps.apache.org by "bchapuis (via GitHub)" <gi...@apache.org> on 2024/04/16 06:57:08 UTC
[I] Add support for Overturemap parquet files [incubator-baremaps]
bchapuis opened a new issue, #849:
URL: https://github.com/apache/incubator-baremaps/issues/849
https://github.com/OvertureMaps/data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "bchapuis (via GitHub)" <gi...@apache.org>.
bchapuis commented on issue #849:
URL: https://github.com/apache/incubator-baremaps/issues/849#issuecomment-2113259112
Yes, I think the suggestion of @Drabble to look into drill is a good idea. We can probably either use it or get inspiration from it for our own implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "bchapuis (via GitHub)" <gi...@apache.org>.
bchapuis commented on issue #849:
URL: https://github.com/apache/incubator-baremaps/issues/849#issuecomment-2143547311
Yes, the plan sounds good and can probably be addressed with multiple PRs. Maybe we can skip step 4 or use views instead of materialized views. As the daylight distribution with soon be deprecated and replaced by overturemaps, an idea could be to copy the daylight directory and use it as a basis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "bchapuis (via GitHub)" <gi...@apache.org>.
bchapuis commented on issue #849:
URL: https://github.com/apache/incubator-baremaps/issues/849#issuecomment-2088415712
@sebr72 as discussed, I'm not really satisfied with my current experiment in the overturemap branch. The geoparquet format contains semi structured data which require some changes in the DataTable abstraction. Also, it requires a deep understanding of the geoparquet format.
One avenue (probably the best) could be to use the parser available in sedona (the project is written in scala):
https://sedona.apache.org/latest-snapshot/tutorial/sql/#__tabbed_9_2
Another avenue could be to build upon my throw-away overturemaps branch, but I'm not sure about the effort needed to have something robust.
In both cases, adding parquet or sedona will result in a lot of new dependencies (hadoop, spark).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "bchapuis (via GitHub)" <gi...@apache.org>.
bchapuis commented on issue #849:
URL: https://github.com/apache/incubator-baremaps/issues/849#issuecomment-2088641794
@sebr72 There may also be a third option which is to rely on parquet support in postgresql. I have no experience with this extention.
https://github.com/adjust/parquet_fdw
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "bchapuis (via GitHub)" <gi...@apache.org>.
bchapuis commented on issue #849:
URL: https://github.com/apache/incubator-baremaps/issues/849#issuecomment-2139180343
@sebr72 @Drabble I will merge the current PR and organize the git history to have three separated commits with our individual contributions. For the following tasks, I suggest we make individual PRs and split the work more clearly.
- [ ] Cleanup the sonar problems when it makes sense (@sebr72 ).
- [ ] Add some unit tests to the GeoParquetReader (@sebr72 ).
- [ ] Define a better organization for the packages and classes of the geoparquet module (data, hadoop, etc.).
- [ ] Improve the allocation of objects in the GeoParquet reader (list for each field, wrapper for each value, etc.)
- [ ] The high level abstration (DataTable, DataColumn, etc.) needs support for nested data structures such as groups in geoparquet and json in other data formats (@bchapuis will make a first refactoring pass).
- [ ] Improve the abstraction of the GeoParquetGroup (the current version uses internal classes, and the getters/setters were quickly definined to have an end-to-end example, etc.) - after doing a pass on the DataTable abstraction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "Drabble (via GitHub)" <gi...@apache.org>.
Drabble commented on issue #849:
URL: https://github.com/apache/incubator-baremaps/issues/849#issuecomment-2143506193
@bchapuis Great job on the pull request! I will look at your new one for nested groups.
I would be really interested in making an example to go from Overture data on S3 to serving MVT to a Maputnik frontend.
I think this would mean:
1 Fix the code to be able to use a S3 url directly. E.g. s3a://overturemaps-us-west-2/release/2024-05-16-beta.0/theme=admins/type=*/*
2. Use the GeoParquetDataTable to write Overture data into Postgresql using a ProjectionTransformer to go from EPSG:4326 to EPSG:3857
3. Create a geospatial index for the geometry column
4. Create a materialised view to group the columns into a TAGS jsonb field and maybe simplifications for different zoom levels
5. Make a simple style.json and tileset.json to serve the data
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "fgravin (via GitHub)" <gi...@apache.org>.
fgravin commented on issue #849:
URL: https://github.com/apache/incubator-baremaps/issues/849#issuecomment-2167622865
Congratz guys :tada:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "Drabble (via GitHub)" <gi...@apache.org>.
Drabble commented on issue #849:
URL: https://github.com/apache/incubator-baremaps/issues/849#issuecomment-2167596215
We have a basic support for Overture maps now. Should we consider this issue closed and raise more issues for further improvements to the Overture maps library?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "bchapuis (via GitHub)" <gi...@apache.org>.
bchapuis closed issue #849: Add support for Overturemap parquet files
URL: https://github.com/apache/incubator-baremaps/issues/849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "sebr72 (via GitHub)" <gi...@apache.org>.
sebr72 commented on issue #849:
URL: https://github.com/apache/incubator-baremaps/issues/849#issuecomment-2112661390
@bchapuis I had a look at Sedona and I highlight the following:
1. Large project mainly relying on Spark or Flink (large project themselves)
2. The java examples are around Flink which is a lot faster than Spark but it is not directly linked to Geoparquet
3. The geoparquet implementation is in Scala and geared around Spark
4. Combining Spark Sedona and Scala with baremaps will end up in an "expensive" integration for Geoparquet.
I am going to switch to have a look at:
https://github.com/apache/drill/tree/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add support for Overturemap parquet files [incubator-baremaps]
Posted by "bchapuis (via GitHub)" <gi...@apache.org>.
bchapuis commented on issue #849:
URL: https://github.com/apache/incubator-baremaps/issues/849#issuecomment-2139411561
@sebr72 @Drabble I merged the changes and we can now continue with individual PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@baremaps.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org