You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by "jwass (via GitHub)" <gi...@apache.org> on 2024/03/29 14:42:15 UTC

[I] Writing to multiple GeoParquet files will not output _metadata [sedona]

jwass opened a new issue, #1296:
URL: https://github.com/apache/sedona/issues/1296

   ## Expected behavior
   
   When writing out a GeoParquet dataframe that results in multiple files, the _metadata summary file will not be created when configured to do so.
   
   ```
   import sedona
   from sedona.spark import *
   sedona = SedonaContext.create(spark)
   
   print("spark version: {}".format(spark.version))
   print("sedona version: {}".format(sedona.version))
   spark.conf.set("parquet.summary.metadata.level", "ALL")
   
   def write_geoparquet(df, path):
       df.write.format("geoparquet") \
           .option("geoparquet.version", "1.0.0") \
           .option("geoparquet.crs", "") \
           .option("compression", "zstd") \
           .option("parquet.block.size", 16 * 1024 * 1024) \
           .option("maxRecordsPerFile", 10000000) \
           .mode("overwrite").save(path)
   
   df = sedona.read.format("geoparquet").option("mergeSchema", "true").load(input_path)
   write_geoparquet(df, output_path)
   ```
   
   If the number of records exceeds maxRecordsPerFile so that more than one file is written, the `_metadata` and `_common_metadata` files will not be written. When there are fewer records that only one file is written, then `_metadata` and `_common_metadata` will be created.
   
   However if I change the above to write parquet instead of geoparquet:
   def write_parquet(df, path):
       df.write.format("parquet") \
           .option("compression", "zstd") \
           .option("parquet.block.size", 16 * 1024 * 1024) \
           .option("maxRecordsPerFile", 10000000) \
           .mode("overwrite").save(path)
   
   write_parquet(df, output_path)
   ```
   
   Then `_metadata` and `_common_metadata` will be written even with multiple files. Is there a setting or other way to enable writing the common metadata files?
   
   I'd like to write these files as reading in full datasets from pyarrow or others will not need to fully scan all files which can be time-consuming for large datasets.
   
   ## Settings
   
   Sedona version = 3.4.1
   Apache Spark version = 3.4.1
   
   Environment = Databricks
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Writing to multiple GeoParquet files will not output _metadata [sedona]

Posted by "jwass (via GitHub)" <gi...@apache.org>.
jwass commented on issue #1296:
URL: https://github.com/apache/sedona/issues/1296#issuecomment-2035500692

   @Kontinuation Thanks. I think it would be totally fine to leave off the `geo` metadata in the combined `_metadata` and/or `_common_metadata` files - as long as it is still present in the individual geoparquet files.
   
   Since GeoParquet doesn't define these I don't think it would be any issue at all - of course in the future it may standardize on a definition but I think for now it'll only be used for row group filtering to the geo metadata is not needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Writing to multiple GeoParquet files will not output _metadata [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu closed issue #1296: Writing to multiple GeoParquet files will not output _metadata
URL: https://github.com/apache/sedona/issues/1296


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Writing to multiple GeoParquet files will not output _metadata [sedona]

Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on issue #1296:
URL: https://github.com/apache/sedona/issues/1296#issuecomment-2029885038

   The `geo` metadata in the parquet footers may not be the same for all written geoparquet files, especially the bbox field, this makes the default parquet footer metadata merging process fail with the following exception:
   
   ```
   java.lang.RuntimeException: could not merge metadata: key geo has conflicting values: [{"version":"1.0.0","primary_column":"geom","columns":{"geom":{"encoding":"WKB","geometry_types":["Polygon"],"bbox":[1.0,1.0,9998.0,9998.0],"crs":null}}}, {"version":"1.0.0","primary_column":"geom","columns":{"geom":{"encoding":"WKB","geometry_types":["Polygon"],"bbox":[0.0,0.0,10000.0,10000.0],"crs":null}}}]
   	at org.apache.parquet.hadoop.metadata.StrictKeyValueMetadataMergeStrategy.merge(StrictKeyValueMetadataMergeStrategy.java:36)
   	at org.apache.parquet.hadoop.metadata.GlobalMetaData.merge(GlobalMetaData.java:106)
   	at org.apache.parquet.hadoop.ParquetFileWriter.mergeFooters(ParquetFileWriter.java:1451)
   	at org.apache.parquet.hadoop.ParquetFileWriter.mergeFooters(ParquetFileWriter.java:1422)
   	at org.apache.parquet.hadoop.ParquetFileWriter.writeMetadataFile(ParquetFileWriter.java:1383)
   	at org.apache.parquet.hadoop.ParquetOutputCommitter.writeMetaDataFile(ParquetOutputCommitter.java:84)
   	at org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:50)
   	at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:192)
   ```
   
   We have to implement an output committer for GeoParquet to merge `geo` metadata properly. If your usecase do not need to read the geo metadata from _common_metadata or _metadata file, we can simply ignore geo metadata when generating such files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org