You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by GitBox <gi...@apache.org> on 2021/08/22 20:11:14 UTC
[GitHub] [incubator-sedona] swamirishi opened a new pull request #536: [SEDONA-36] Parquet reader & Writers
swamirishi opened a new pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536
## Is this PR related to a proposed Issue?
https://issues.apache.org/jira/projects/SEDONA/issues/SEDONA-36
https://issues.apache.org/jira/projects/SEDONA/issues/SEDONA-14
The issue is realted to serializing & deserializing Spatial RDDs into Parquet files with a schema
## What changes were proposed in this PR?
Serializing & Deserializing issues in Spatial RDDs into Parquet Files.
## How was this patch tested?
Unit Test cases & Integration tests are in place. Not touching existing code flow thus previous functionalities should not get affected
## Did this PR include necessary documentation updates?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] geoHeil commented on pull request #536: Sedona 36- Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
geoHeil commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-895825848
Does this follow the https://github.com/geopandas/geo-arrow-spec arrow geo spec? Will it be possible to store metadata about the used projection (CRS) inside the parquet file (like geo-package). I believe geo-pandas is doing this as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi removed a comment on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi removed a comment on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-903323992
> > Would this PR support multiple geometry columns in the parquet file? But one designated one for calculating the statistics at the row-group level?
>
> Currently only one single geometry column per record. We can go ahead and support such features in later versions. The stats are stored on Schemas Column Path. So for multiple Columns the Stats stored would be stored on different paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695577641
##########
File path: core/src/main/java/org/apache/sedona/core/formatMapper/ParquetReader.java
##########
@@ -0,0 +1,38 @@
+package org.apache.sedona.core.formatMapper;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.formatMapper.parquet.ParquetFormatMapper;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.io.parquet.ParquetFileReader;
+import org.apache.sedona.core.spatialRDD.SpatialRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.locationtech.jts.geom.Geometry;
+import org.locationtech.jts.geom.GeometryFactory;
+import org.locationtech.jts.geom.LineString;
+import org.locationtech.jts.geom.Polygon;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ParquetReader extends RddReader {
+ public static <T extends Geometry> SpatialRDD<T> createSpatialRDD(JavaRDD rawRDD,
+ ParquetFormatMapper<T> formatMapper,
+ GeometryType geometryType) {
+ SpatialRDD spatialRDD = new SpatialRDD<T>(geometryType);
+ spatialRDD.rawSpatialRDD = rawRDD.mapPartitions(formatMapper);
+ return spatialRDD;
+ }
+
+ public static <T extends Geometry> SpatialRDD<T> readToGeometryRDD(JavaSparkContext sc,
+ List<String> inputPath,
Review comment:
The input Paths can be a path regex. I have changed the testcase accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-903283881
@jiayuasu @Imbruced @netanel246 @Sarwat Did you take a look at the PR? I would like to get this merged as soon as possible so that I can start working on the second phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi edited a comment on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi edited a comment on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-903323728
> Would this PR support multiple geometry columns in the parquet file? But one designated one for calculating the statistics at the row-group level?
@geoHeil Currently only one single geometry column per record. We can go ahead and support such features in later versions. The stats are stored on Schemas Column Path. So for multiple Columns the Stats stored would be stored on different paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu merged pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu merged pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu edited a comment on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu edited a comment on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-905314053
@swamirishi
Thank you again for your patience! I traveled back to WSU and the new semester just started. So I was kind of overwhelmed by many things on my plate. Now I am ready to work with you!
I personally are very excited about this PR but I think the reason why several committers hesitate to approve this PR is that
1. it does not have a detailed proposal to explain the proposed file structure and how it works. We know the overall idea of having a geospatial Parquet file and it is an awesome idea for sure. But it is unclear to us how this is achieved in this PR.
2. it contains 44 file changes which are too many to be reviewed by a human... It touches several critical places in Sedona. For instance, it adds a new module called Parquet, adds some Sedona exceptions, change some dependencies. More importantly, it changes the core/SpatialRDD.java file which will affect many places.
Therefore, I believe here are some action items
1. would you please add a Sedona website doc in this PR to explain
* The usage of Sedona Parquet reader and writer
* The structure of the proposed geo Parquet file, especially the metadata structure
* the algorithm: how it skips irrelevant data chunks by comparing the spatial query predicate and Parquet metadata
You can put this doc at https://sedona.apache.org/tutorial/ as a programming guide for geospatial Parquet
2. for the file changes themselves
* please fix the comments raised by @Imbruced
* Please address my comment for SpatialRDD change
* please remove all whitespace file changes, and changes that just include "re-formatting the code"
Once these changes + doc are done and we fully understand what this PR is doing exactly, I will approve this PR. Let the customers give it a try. I believe once this PR is accepted, your follow-up PRs for the Parquet file will quickly pass our review.
Thank you again for your contribution. We really appreciate your contribution!
Jia Yu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] Imbruced commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
Imbruced commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r691363414
##########
File path: pom.xml
##########
@@ -171,6 +209,12 @@
<artifactId>gt-coverage</artifactId>
<version>${geotools.version}</version>
<scope>${dependency.scope}</scope>
+ <exclusions>
Review comment:
guava ;) always causes issue with dependencies.
##########
File path: core/src/main/java/org/apache/sedona/core/geometryObjects/schema/PolygonSchema.java
##########
@@ -0,0 +1,37 @@
+package org.apache.sedona.core.geometryObjects.schema;
+
+import com.google.common.collect.Lists;
+import org.apache.sedona.core.io.avro.constants.AvroConstants;
+import org.apache.sedona.core.io.avro.schema.*;
+import org.apache.sedona.core.io.avro.utils.AvroUtils;
+import org.apache.sedona.core.utils.SedonaUtils;
+
+/**
+ * Polygon Schema class representing AvroSchema of Polygon Geometry
+ */
+public class PolygonSchema extends RecordSchema {
+ public static final String POLYGON = "polygon";
+ public static final String EXTERIOR_RING = "ex";
+ public static final String HOLES = "holes";
+
+ private PolygonSchema() {
+ super(AvroConstants.SEDONA_NAMESPACE,POLYGON,
+ Lists.newArrayList(new Field(EXTERIOR_RING, CoordinateArraySchema.getSchema()),
+ new Field(HOLES, new UnionSchema(
+ new ArraySchema(CoordinateArraySchema.getSchema()),
+ new SimpleSchema(AvroConstants.PrimitiveDataType.NULL)))));
Review comment:
is better approach to save null or empty list ? WDYT ?
##########
File path: pom.xml
##########
@@ -351,6 +429,25 @@
</execution>
</executions>
</plugin>
+<!-- <plugin>-->
Review comment:
Can we remove commented blocks ?
##########
File path: core/src/main/java/org/apache/sedona/core/formatMapper/parquet/ParquetFormatMapper.java
##########
@@ -0,0 +1,215 @@
+package org.apache.sedona.core.formatMapper.parquet;
+
+import org.apache.avro.generic.GenericArray;
+import org.apache.avro.generic.GenericContainer;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.log4j.Logger;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.formatMapper.FormatMapper;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.geometryObjects.schema.CircleSchema;
+import org.apache.sedona.core.geometryObjects.schema.CoordinateSchema;
+import org.apache.sedona.core.geometryObjects.schema.PolygonSchema;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.zookeeper.Op;
+import org.locationtech.jts.geom.*;
+
+import java.io.Serializable;
+import java.util.*;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.StreamSupport;
+
+/**
+ * ParquetFormatMapper class
+ * Responsible for converting Avro-Parquet Records into Geometry Objects
+ * @param <T> Geometry Class Type
+ */
+public class ParquetFormatMapper<T extends Geometry> implements Serializable, FlatMapFunction<Iterator<GenericRecord>, T> {
+ final static Logger logger = Logger.getLogger(FormatMapper.class);
+ protected GeometryFactory factory = new GeometryFactory();
+ private GeometryType geometryType;
+ private final String geometryColumn;
+ private final List<String> userColumns;
+
+ /**
+ *
+ * @param geometryType
+ * @param geometryColumn
+ * @param userColumns
+ */
+ public ParquetFormatMapper(GeometryType geometryType, String geometryColumn, List<String> userColumns) {
+ this.geometryType = geometryType;
+ this.geometryColumn = geometryColumn;
+ this.userColumns = userColumns;
+ }
+
+ /**
+ * Gets Coordinate from Avro Record
+ * @param record Avro Record
+ * @return Coordinate
+ */
+ private static Coordinate getCoordinate(GenericRecord record) {
+ Double x = (Double) record.get(CoordinateSchema.X_COORDINATE);
+ Double y = (Double) record.get(CoordinateSchema.Y_COORDINATE);
+ return new Coordinate(x, y);
+ }
+
+ /**
+ * Gets Coordinate from Avro based Record
+ * @param val
+ * @return Coordinate
+ */
+ private static Coordinate getCoordinate(GenericContainer val) {
+ if (val instanceof GenericRecord) {
+ return getCoordinate((GenericRecord) val);
+ }
+ return getCoordinate((GenericArray) val);
+ }
+ /**
+ * Gets Coordinate from Avro based Array based on the first index
+ * @param array
+ * @return Coordinate
+ */
+ private static Coordinate getCoordinate(GenericArray array) {
+ return getCoordinate((GenericRecord) array.get(0));
+ }
+
+ /**
+ * Gets Coordinate Array from Avro Record Array
+ * @param array
+ * @return Coordinate Array
+ */
+ private static Coordinate[] getCoordinates(GenericArray array) {
+ Coordinate[] coordinates = new Coordinate[array.size()];
+ for(int i=0;i<array.size();i++){
+ coordinates[i] = getCoordinate((GenericRecord) array.get(i));
+ }
+ return coordinates;
+ }
+
+ /**
+ * Gets Point Geometry from Coordinate
+ * @param coordinate
+ * @return Point
+ */
+ private Point getPoint(Coordinate coordinate) {
+ return factory.createPoint(coordinate);
+ }
+
+ /**
+ * Gets Circle Geometry Object from Avro Record
+ * @param record
+ * @return Circle
+ */
+ private Circle getCircle(GenericRecord record) {
+ Coordinate center = getCoordinate((GenericRecord) record.get(CircleSchema.CENTER));
+ Double radius = (Double) record.get(CircleSchema.RADIUS);
+ return new Circle(getPoint(center), radius);
+ }
+ /**
+ * Gets Polygon Geometry Object from Avro Record
+ * @param record
+ * @return Polygon
+ */
+ private Polygon getPolygon(GenericRecord record) {
+ LinearRing exteriorRing =
+ factory.createLinearRing(getCoordinates((GenericArray) record.get(PolygonSchema.EXTERIOR_RING)));
+ Optional<GenericArray> holes = Optional.ofNullable((GenericArray) record.get(PolygonSchema.HOLES));
+ List<LinearRing> interiorRings = holes.map(arr->IntStream.range(0, arr.size())
+ .mapToObj(arr::get)
+ .map(hole -> getCoordinates((GenericArray) (hole)))
+ .map(c -> factory.createLinearRing(c))
+ .collect(Collectors.toList())).orElse(Collections.EMPTY_LIST);
+
+ return factory.createPolygon(exteriorRing, (LinearRing[]) interiorRings.toArray(new LinearRing[interiorRings.size()]));
+ }
+
+ /**
+ * Gets Geometry Object of Geometry Type from Coordinate Array
+ * @param coordinates CoordinateArray
+ * @param type geometryType
+ * @return Geometry Object
+ */
+ private Geometry getGeometry(Coordinate[] coordinates, GeometryType type) {
+ Geometry geometry = null;
+ switch (type) {
+ case POINT:
+ geometry = factory.createPoint(coordinates[0]);
+ break;
+ case POLYGON:
+ geometry = factory.createPolygon(coordinates);
Review comment:
what if polygon has holes, does that change something ?
##########
File path: core/src/main/java/org/apache/sedona/core/io/parquet/ParquetFileReader.java
##########
@@ -0,0 +1,60 @@
+package org.apache.sedona.core.io.parquet;
+
+import com.google.common.collect.Lists;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.LongWritable;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.parquet.avro.AvroParquetInputFormat;
+import org.apache.parquet.cli.util.Expressions;
+import org.apache.parquet.cli.util.Schemas;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.sedona.core.constants.SedonaConstants;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import scala.Tuple2;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+public class ParquetFileReader {
+ /**
+ * Reads Parquet File with given geometry Column and the relevant User columns
+ * @param sc
+ * @param geometryColumn
+ * @param userColumns
+ * @param inputPaths
+ * @return Avro Record RDD which needs to be deserialized into a GeometryRDD
+ * @throws IOException
+ */
+ public static JavaRDD<GenericRecord> readFile(JavaSparkContext sc,
+ String geometryColumn,
+ List<String> userColumns,
+ String... inputPaths) throws IOException {
+ final Job job = Job.getInstance(sc.hadoopConfiguration());
Review comment:
is that mean that current solution works only in hadoop ecosystem ?
##########
File path: core/src/main/java/org/apache/sedona/core/formatMapper/ParquetReader.java
##########
@@ -0,0 +1,38 @@
+package org.apache.sedona.core.formatMapper;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.formatMapper.parquet.ParquetFormatMapper;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.io.parquet.ParquetFileReader;
+import org.apache.sedona.core.spatialRDD.SpatialRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.locationtech.jts.geom.Geometry;
+import org.locationtech.jts.geom.GeometryFactory;
+import org.locationtech.jts.geom.LineString;
+import org.locationtech.jts.geom.Polygon;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ParquetReader extends RddReader {
+ public static <T extends Geometry> SpatialRDD<T> createSpatialRDD(JavaRDD rawRDD,
+ ParquetFormatMapper<T> formatMapper,
+ GeometryType geometryType) {
+ SpatialRDD spatialRDD = new SpatialRDD<T>(geometryType);
+ spatialRDD.rawSpatialRDD = rawRDD.mapPartitions(formatMapper);
+ return spatialRDD;
+ }
+
+ public static <T extends Geometry> SpatialRDD<T> readToGeometryRDD(JavaSparkContext sc,
+ List<String> inputPath,
Review comment:
Is it possible to specify parent path in case of partitioned files ? Can we add additional method ?
##########
File path: core/src/main/java/org/apache/sedona/core/formatMapper/ParquetReader.java
##########
@@ -0,0 +1,38 @@
+package org.apache.sedona.core.formatMapper;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.formatMapper.parquet.ParquetFormatMapper;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.io.parquet.ParquetFileReader;
+import org.apache.sedona.core.spatialRDD.SpatialRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.locationtech.jts.geom.Geometry;
+import org.locationtech.jts.geom.GeometryFactory;
+import org.locationtech.jts.geom.LineString;
+import org.locationtech.jts.geom.Polygon;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ParquetReader extends RddReader {
+ public static <T extends Geometry> SpatialRDD<T> createSpatialRDD(JavaRDD rawRDD,
+ ParquetFormatMapper<T> formatMapper,
+ GeometryType geometryType) {
+ SpatialRDD spatialRDD = new SpatialRDD<T>(geometryType);
+ spatialRDD.rawSpatialRDD = rawRDD.mapPartitions(formatMapper);
+ return spatialRDD;
+ }
+
+ public static <T extends Geometry> SpatialRDD<T> readToGeometryRDD(JavaSparkContext sc,
Review comment:
Is that final API for the user ? Is that possible to read the geometry column and its type from parquet metadata ?
##########
File path: core/src/test/java/org/apache/sedona/core/io/parquet/ParquetFileReaderTest.java
##########
@@ -0,0 +1,87 @@
+package org.apache.sedona.core.io.parquet;
+
+import com.clearspring.analytics.util.Lists;
+import org.apache.commons.io.FileUtils;
+import org.apache.sedona.core.TestBase;
+import org.apache.sedona.core.enums.FileDataSplitter;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.exceptions.SedonaException;
+import org.apache.sedona.core.formatMapper.ParquetReader;
+import org.apache.sedona.core.formatMapper.WktReader;
+import org.apache.sedona.core.formatMapper.WktReaderTest;
+import org.apache.sedona.core.spatialRDD.PointRDD;
+import org.apache.sedona.core.spatialRDD.SpatialRDD;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.locationtech.jts.geom.Point;
+import org.locationtech.jts.util.Assert;
+
+import java.io.IOException;
+import java.nio.file.FileSystem;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class ParquetFileReaderTest extends TestBase {
+ private static String filePath;
+ @BeforeClass
+ public static void onceExecutedBeforeAll()
+ throws IOException
+ {
+ initialize(ParquetFileReaderTest.class.getName());
+ filePath = ParquetFileReaderTest.class.getClassLoader().getResource("crs-test-point.csv").getPath();
+ }
+
+ @AfterClass
+ public static void tearDown()
+ throws Exception
+ {
+ sc.stop();
+ }
+
+ @Before
+ public void delete() throws IOException {
+ if(testOutputPathDir.exists()){
+ FileUtils.deleteDirectory(testOutputPathDir);
+ }
+ }
+
+ @Test
+ public void testParquetReader() throws IOException, SedonaException {
+ PointRDD pointRDD = new PointRDD(sc, filePath, 0, FileDataSplitter.CSV, false, 1);
+ pointRDD.saveAsParquet(sc,
+ "p",
+ Lists.newArrayList(),
+ testOutputPathDir.getAbsolutePath(),
+ "test.namespace",
+ "name");
+ List<Point> l1 = pointRDD.getRawSpatialRDD().collect().stream().collect(Collectors.toList());
+ Collections.sort(l1, new Comparator<Point>() {
+ @Override
+ public int compare(Point o1, Point o2) {
+ return o1.compareTo(o2);
+ }
+ });
+
+ SpatialRDD<Point> pointRDD1 = ParquetReader.readToGeometryRDD(sc,
+ Arrays.stream(testOutputPathDir.listFiles())
Review comment:
is that possible to load parquets from parent directory ex.
test
- *.parquet
- *.parquet
- *.parquet
- *.parquet
Can we read this like readToGeometryRDD(..., test) instead of listing all files ?
##########
File path: core/src/main/java/org/apache/sedona/core/geometryObjects/schema/CoordinateSchema.java
##########
@@ -0,0 +1,39 @@
+package org.apache.sedona.core.geometryObjects.schema;
+
+import com.google.common.collect.Lists;
+import org.apache.sedona.core.io.avro.constants.AvroConstants;
+import org.apache.sedona.core.io.avro.schema.Field;
+import org.apache.sedona.core.io.avro.schema.SimpleSchema;
+import org.apache.sedona.core.io.avro.schema.RecordSchema;
+import org.apache.sedona.core.utils.SedonaUtils;
+
+public class CoordinateSchema extends RecordSchema {
+ public static final String X_COORDINATE = "x";
+ public static final String Y_COORDINATE = "y";
+ public static final String COORDINATE = "coordinate";
+
+ private CoordinateSchema() {
+ super(AvroConstants.SEDONA_NAMESPACE,COORDINATE,
+ Lists.newArrayList(new Field(X_COORDINATE, new SimpleSchema(AvroConstants.PrimitiveDataType.DOUBLE)),
+ new Field(Y_COORDINATE, new SimpleSchema(AvroConstants.PrimitiveDataType.DOUBLE))));
+ }
+
+ private static CoordinateSchema schema;
+
+ public static CoordinateSchema getSchema(){
+ if(SedonaUtils.isNull(schema)){
+ synchronized (CoordinateSchema.class){
+ if(SedonaUtils.isNull(schema)){
+ schema = new CoordinateSchema();
+ }
+ }
+ }
+ return schema;
+ }
+
+// public CoordinateSchema(String name, String namespace) {
Review comment:
can we remove that code ?
##########
File path: core/src/main/java/org/apache/sedona/core/showcase/Example.java
##########
@@ -52,99 +52,99 @@
public class Example
implements Serializable
{
-
+
/**
* The sc.
*/
public static JavaSparkContext sc;
-
+
/**
* The geometry factory.
*/
static GeometryFactory geometryFactory;
-
+
/**
* The Point RDD input location.
*/
static String PointRDDInputLocation;
-
+
/**
* The Point RDD offset.
*/
static Integer PointRDDOffset;
-
+
/**
* The Point RDD num partitions.
*/
static Integer PointRDDNumPartitions;
-
+
/**
* The Point RDD splitter.
*/
static FileDataSplitter PointRDDSplitter;
-
+
/**
* The Point RDD index type.
*/
static IndexType PointRDDIndexType;
-
+
Review comment:
I dont like that kind of changes ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-907922385
@swamirishi
Can you also address my comments? Then we can move to accept this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] geoHeil commented on a change in pull request #536: Sedona 36- Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
geoHeil commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r685797212
##########
File path: core/src/main/java/org/apache/sedona/core/io/avro/utils/AvroUtils.java
##########
@@ -0,0 +1,263 @@
+package org.apache.sedona.core.io.avro.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.ImmutableMap;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericArray;
+import org.apache.avro.generic.GenericContainer;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.exceptions.SedonaException;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.geometryObjects.schema.CircleSchema;
+import org.apache.sedona.core.geometryObjects.schema.CoordinateArraySchema;
+import org.apache.sedona.core.geometryObjects.schema.CoordinateSchema;
+import org.apache.sedona.core.geometryObjects.schema.PolygonSchema;
+import org.apache.sedona.core.io.avro.SchemaUtils;
+import org.apache.sedona.core.io.avro.constants.AvroConstants;
+import org.apache.sedona.core.io.avro.schema.Field;
+import org.apache.sedona.core.io.avro.schema.RecordSchema;
+import org.locationtech.jts.geom.*;
+
+import java.util.*;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+public class AvroUtils {
+ public static String getNestedNamespace(String namespace, String... name) {
+
+ return String.join(AvroConstants.DOT, namespace, String.join(AvroConstants.DOT, name));
+ }
+
+
+ public static GenericRecord getRecord(Schema avroSchema, Map<String, Object> value) throws SedonaException {
+ GenericRecord record = new GenericData.Record(avroSchema);
+ for (Schema.Field field : avroSchema.getFields()) {
+ record.put(field.name(), getRecord(field.schema(), value.get(field.name())));
+ }
+ return record;
+ }
+
+ public static GenericArray getArray(Schema avroSchema, Collection<Object> values) throws SedonaException {
+
+ List<Object> vals = Lists.newArrayList();
+ for (Object val : values) {
+ vals.add(getRecord(avroSchema.getElementType(), val));
+ }
+ return new GenericData.Array<>(avroSchema, vals);
+ }
+
+ public static Object getUnion(Schema avroSchema, Object value) throws SedonaException{
+ for(Schema typeSchema:avroSchema.getTypes()){
+ try{
+ return getRecord(typeSchema,value);
+ }catch (SedonaException e){
+
+ }
+ }
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()));
+ }
+
+ public static int getInteger(Object value){
+ return Integer.parseInt(value.toString());
+ }
+
+ public static long getLong(Object value){
+ return Long.parseLong(value.toString());
+ }
+
+ public static double getDouble(Object value){
+ return Double.parseDouble(value.toString());
+ }
+
+ public static float getFloat(Object value){
+ return Float.parseFloat(value.toString());
+ }
+
+
+
+ public static Object getRecord(Schema avroSchema, Object value) throws SedonaException {
+ try {
+ if (value == null) {
+ if(Schema.Type.NULL.equals(avroSchema.getType())){
+ return null;
+ }
+ if(!Schema.Type.UNION.equals(avroSchema.getType())){
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()));
+ }
+ }
+ if(value instanceof GenericContainer){
+ return value;
+ }
+
+ if (avroSchema == null) {
+ throw new SedonaException("Null Schema Provided");
+ }
+ switch (avroSchema.getType()) {
+ case RECORD:
+ return getRecord(avroSchema, (Map<String, Object>) value);
+ case ARRAY:
+ return getArray(avroSchema, (Collection<Object>) value);
+ case UNION:
+ return getUnion(avroSchema, value);
+ case INT:
+ return getInteger(value);
+ case LONG:
+ return getLong(value);
+ case FLOAT:
+ return getFloat(value);
+ case DOUBLE:
+ return getDouble(value);
+ default:
+ return value;
+ }
+ }catch (SedonaException e){
+ throw e;
+ } catch (Exception e) {
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()),e);
+ }
+ }
+
+ public static Field getField(String columnName, GeometryType geometryType) throws SedonaException {
+ org.apache.sedona.core.io.avro.schema.Schema schema;
+ switch (geometryType) {
+ case CIRCLE: {
+ schema = CircleSchema.getSchema();
+ break;
+ }
+ case POINT: {
+ schema = CoordinateSchema.getSchema();
+ break;
+ }
+ case RECTANGLE:
+ case POLYGON: {
+ schema = PolygonSchema.getSchema();
+ break;
+ }
+ case LINESTRING: {
+ schema = CoordinateArraySchema.getSchema();
+ break;
+ }
+ default:
+ throw new SedonaException("Geometry Type not Supported");
+ }
+ return new Field(columnName, schema);
+ }
+
+ public static Schema getGeometryRecordSchema(String namespace,
+ String name,
+ Field geometryField,
+ Field userColumn) throws SedonaException {
+ return getGeometryRecordSchema(namespace, name, geometryField, Lists.newArrayList(userColumn));
+ }
+
+ public static Schema getGeometryRecordSchema(String namespace,
+ String name,
+ Field geometryField,
+ List<Field> userColumns) throws SedonaException {
+ List<Field> columns = Lists.newArrayList(geometryField);
+ columns.addAll(userColumns);
+ RecordSchema schema = new RecordSchema(namespace,name, columns);
+ return SchemaUtils.SchemaParser.getSchema(schema.getDataType());
+ }
+
+ public static Map<String, Object> getMapFromCircle(Circle circle) {
+ Coordinate center = circle.getCoordinate();
+ Double radius = circle.getRadius();
+ return ImmutableMap.of(CircleSchema.CENTER, getMapFromCoordinate(center), CircleSchema.RADIUS, radius);
+
+ }
+
+ public static Map<String, Double> getMapFromCoordinate(Coordinate coordinate) {
+ return ImmutableMap.of(CoordinateSchema.X_COORDINATE,
+ coordinate.x,
+ CoordinateSchema.Y_COORDINATE,
+ coordinate.y);
+ }
+
+ public static Collection<Map<String, Double>> getCollectionFromCoordinates(Coordinate[] coordinates) {
+ return Arrays.stream(coordinates).map(AvroUtils::getMapFromCoordinate).collect(Collectors.toList());
+ }
+
+ public static Collection<Map<String, Double>> getCollectionFromLineString(LineString lineString) {
+ return getCollectionFromCoordinates(lineString.getCoordinates());
+ }
+
+ public static Map<String, Double> getMapFromPoint(Point point) {
+ return getMapFromCoordinate(point.getCoordinate());
+ }
+
+ public static Map<String, Object> getMapFromPolygon(Polygon polygon) {
+ return ImmutableMap.of(PolygonSchema.EXTERIOR_RING,
+ getCollectionFromCoordinates(polygon.getExteriorRing().getCoordinates()),
+ PolygonSchema.HOLES,
+ IntStream.range(0, polygon.getNumInteriorRing())
+ .mapToObj(polygon::getInteriorRingN)
+ .map(linearRing -> getCollectionFromCoordinates(linearRing.getCoordinates()))
+ .collect(Collectors.toList()));
+ }
+
+ public static GenericRecord getRecordFromGeometry(Schema schema,
+ String geometryColumnName,
+ String userColumn,
+ Object geometryData,
+ Object userData) throws SedonaException {
+ return (GenericRecord) getRecord(schema,
+ ImmutableMap.of(geometryColumnName, geometryData, userColumn, userData));
+ }
+
+ public static GenericRecord getRecordFromGeometry(Schema schema,
+ String geometryColumnName,
+ Object geometryData,
+ Optional<Map<String, Object>> userData) throws SedonaException {
+ return (GenericRecord) getRecord(schema,
+ (Object) new ImmutableMap.Builder<>().putAll(userData.orElse(Collections.EMPTY_MAP))
+ .put(geometryColumnName, geometryData)
+ .build());
+ }
+
+ public static Schema getSchema(GeometryType geometryType,
+ String geometryColumnName,
+ List<Field> userColumns,
+ String namespace,
+ String name) throws SedonaException {
+ Field geometryField = getField(geometryColumnName, geometryType);
+ return getGeometryRecordSchema(namespace, name, geometryField, userColumns);
+ }
+
+ public static GenericRecord getRecord(Geometry geometry,
+ GeometryType type,
+ String geometryColumnName,
+ Schema avroSchema) throws SedonaException {
+ Object geometryData;
+ switch (type) {
+ case CIRCLE: {
+ geometryData = getMapFromCircle((Circle) geometry);
+ break;
+ }
+ case LINESTRING: {
+ geometryData = getCollectionFromLineString((LineString) geometry);
+ break;
+ }
+ case POINT: {
+ geometryData = getMapFromPoint((Point) geometry);
+ break;
+ }
+ case POLYGON:
Review comment:
What about MultiPolygons? Wouldn`t it be great if complex geometries are supported natively?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-903323992
> > Would this PR support multiple geometry columns in the parquet file? But one designated one for calculating the statistics at the row-group level?
>
> Currently only one single geometry column per record. We can go ahead and support such features in later versions. The stats are stored on Schemas Column Path. So for multiple Columns the Stats stored would be stored on different paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695172638
##########
File path: core/src/main/java/org/apache/sedona/core/formatMapper/ParquetReader.java
##########
@@ -0,0 +1,38 @@
+package org.apache.sedona.core.formatMapper;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.formatMapper.parquet.ParquetFormatMapper;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.io.parquet.ParquetFileReader;
+import org.apache.sedona.core.spatialRDD.SpatialRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.locationtech.jts.geom.Geometry;
+import org.locationtech.jts.geom.GeometryFactory;
+import org.locationtech.jts.geom.LineString;
+import org.locationtech.jts.geom.Polygon;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ParquetReader extends RddReader {
+ public static <T extends Geometry> SpatialRDD<T> createSpatialRDD(JavaRDD rawRDD,
+ ParquetFormatMapper<T> formatMapper,
+ GeometryType geometryType) {
+ SpatialRDD spatialRDD = new SpatialRDD<T>(geometryType);
+ spatialRDD.rawSpatialRDD = rawRDD.mapPartitions(formatMapper);
+ return spatialRDD;
+ }
+
+ public static <T extends Geometry> SpatialRDD<T> readToGeometryRDD(JavaSparkContext sc,
+ List<String> inputPath,
Review comment:
Yeah we can have method which can get all files in a directory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-897067279
> @swamirishi Thanks for your great work. This PR sounds interesting to me. However, this seems to be a big PR so we need to have a thorough review for this.
>
> So far, this PR contains many file changes: 47 files changed so far. Can you avoid changing POM files unless it is necessary? Currently, you duplicate the dependencies to different Sedona modules. Please only keep your changes to Sedona core module if possible. This will be helpful for us to review.
> I had to shade Avro Parquet as there is a conflict with the Sernetcdf library. I have moved modules into dependency management. Those are not duplicate now a module would use only the libraries it would require. Thus making the dependency requirement for the project uniform throughout.
> Can you also make your PR pass our CI test?
> Yeah have fixed the issue will push the code. Some issues after rebasing my branch with the master
> @Imbruced @netanel246 @Sarwat Folks, please take a look when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] Imbruced commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
Imbruced commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-901222286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-896078189
@swamirishi Thanks for your contribution. However, currently, Sedona DataFrame can be saved and loaded as Parquet files. I am not sure if we need this function for RDD. We have the adapter to travel between RDD and DataFrame.
Could you elaborate more about the purpose of this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r686123990
##########
File path: core/src/main/java/org/apache/sedona/core/spatialRDD/SpatialRDD.java
##########
@@ -564,7 +588,45 @@ public void saveAsWKT(String outputLocation)
}
}).saveAsTextFile(outputLocation);
}
-
+
+ public void saveAsParquet(JavaSparkContext sc,
+ String geometryColumn,
+ List<Field> userColumns,
+ String outputLocation,
+ String namespace,
+ String name) throws SedonaException, IOException {
+ sc.hadoopConfiguration().setBoolean("parquet.enable.summary-metadata", false);
+ final Job job = Job.getInstance(sc.hadoopConfiguration());
+ GeometryType geometryType = this.getGeometryType();
+ Schema schema = AvroUtils.getSchema(geometryType, geometryColumn, userColumns,namespace,
+ name);
+// ParquetOutputFormat.setWriteSupportClass(job, AvroWriteSupport.class);
+ AvroParquetOutputFormat.setSchema(job, schema);
+ this.rawSpatialRDD.mapPartitionsToPair(new PairFlatMapFunction<Iterator<T>, Void, GenericRecord>() {
+ @Override
+ public Iterator<Tuple2<Void, GenericRecord>> call(Iterator<T> iterator) throws Exception {
+ Schema schema = AvroUtils.getSchema(geometryType, geometryColumn, userColumns, namespace,
+ name);
+ Iterable<T> recordIterable = () -> iterator;
+ try {
+ return StreamSupport.stream(recordIterable.spliterator(), false).map(geometry -> {
+ try {
+ GenericRecord genericRecord =
+ AvroUtils.getRecord(geometry, geometryType, geometryColumn, schema);
Review comment:
We wont be able to store the required Statistics for Predicate Push. Parquet stores page level Aggregate statistics. Thus when we have to do a predicate push or spatial join. Currently this is the phase 1(Serialization & Deserialization from Parquet to Geometry Object). Predicate Push & Join support would be part of Phase 2.
E.g. This is the Meta stats stored in a Parquet File Footer. This is a file storing polygon geometry data
file created_by parquet-mr version 1.12.0 (build db75a6815f2ba1d1ee89d1a90aeb296f1f3a8f20)
file columns 5
file row_groups 1
file rows 1509
row_group 0 size 50818
row_group 0 rows 1509
row_group 0 columns 5
row_group 0 p.ex.array.x type DOUBLE
row_group 0 p.ex.array.x num_values 7545
row_group 0 p.ex.array.x compression UNCOMPRESSED
row_group 0 p.ex.array.x encodings RLE,PLAIN_DICTIONARY
row_group 0 p.ex.array.x compressed_size 25270
row_group 0 p.ex.array.x uncompressed_size 25270
row_group 0 p.ex.array.x stats:min -158.104182
row_group 0 p.ex.array.x stats:max -66.03575
row_group 0 p.ex.array.y type DOUBLE
row_group 0 p.ex.array.y num_values 7545
row_group 0 p.ex.array.y compression UNCOMPRESSED
row_group 0 p.ex.array.y encodings RLE,PLAIN_DICTIONARY
row_group 0 p.ex.array.y compressed_size 25406
row_group 0 p.ex.array.y uncompressed_size 25406
row_group 0 p.ex.array.y stats:min 17.986328
row_group 0 p.ex.array.y stats:max 47.777622
row_group 0 p.holes.array.array.x type DOUBLE
row_group 0 p.holes.array.array.x num_values 1509
row_group 0 p.holes.array.array.x compression UNCOMPRESSED
row_group 0 p.holes.array.array.x encodings RLE,PLAIN
row_group 0 p.holes.array.array.x compressed_size 38
row_group 0 p.holes.array.array.x uncompressed_size 38
row_group 0 p.holes.array.array.y type DOUBLE
row_group 0 p.holes.array.array.y num_values 1509
row_group 0 p.holes.array.array.y compression UNCOMPRESSED
row_group 0 p.holes.array.array.y encodings RLE,PLAIN
row_group 0 p.holes.array.array.y compressed_size 38
row_group 0 p.holes.array.array.y uncompressed_size 38
row_group 0 user_data type BYTE_ARRAY
row_group 0 user_data num_values 1509
row_group 0 user_data compression UNCOMPRESSED
row_group 0 user_data encodings PLAIN_DICTIONARY,BIT_PACKED
row_group 0 user_data compressed_size 66
row_group 0 user_data uncompressed_size 66
row_group 0 user_data stats:min testAttribute123
row_group 0 user_data stats:max testAttribute123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r686124832
##########
File path: core/src/main/java/org/apache/sedona/core/io/avro/utils/AvroUtils.java
##########
@@ -0,0 +1,263 @@
+package org.apache.sedona.core.io.avro.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.ImmutableMap;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericArray;
+import org.apache.avro.generic.GenericContainer;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.exceptions.SedonaException;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.geometryObjects.schema.CircleSchema;
+import org.apache.sedona.core.geometryObjects.schema.CoordinateArraySchema;
+import org.apache.sedona.core.geometryObjects.schema.CoordinateSchema;
+import org.apache.sedona.core.geometryObjects.schema.PolygonSchema;
+import org.apache.sedona.core.io.avro.SchemaUtils;
+import org.apache.sedona.core.io.avro.constants.AvroConstants;
+import org.apache.sedona.core.io.avro.schema.Field;
+import org.apache.sedona.core.io.avro.schema.RecordSchema;
+import org.locationtech.jts.geom.*;
+
+import java.util.*;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+public class AvroUtils {
+ public static String getNestedNamespace(String namespace, String... name) {
+
+ return String.join(AvroConstants.DOT, namespace, String.join(AvroConstants.DOT, name));
+ }
+
+
+ public static GenericRecord getRecord(Schema avroSchema, Map<String, Object> value) throws SedonaException {
+ GenericRecord record = new GenericData.Record(avroSchema);
+ for (Schema.Field field : avroSchema.getFields()) {
+ record.put(field.name(), getRecord(field.schema(), value.get(field.name())));
+ }
+ return record;
+ }
+
+ public static GenericArray getArray(Schema avroSchema, Collection<Object> values) throws SedonaException {
+
+ List<Object> vals = Lists.newArrayList();
+ for (Object val : values) {
+ vals.add(getRecord(avroSchema.getElementType(), val));
+ }
+ return new GenericData.Array<>(avroSchema, vals);
+ }
+
+ public static Object getUnion(Schema avroSchema, Object value) throws SedonaException{
+ for(Schema typeSchema:avroSchema.getTypes()){
+ try{
+ return getRecord(typeSchema,value);
+ }catch (SedonaException e){
+
+ }
+ }
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()));
+ }
+
+ public static int getInteger(Object value){
+ return Integer.parseInt(value.toString());
+ }
+
+ public static long getLong(Object value){
+ return Long.parseLong(value.toString());
+ }
+
+ public static double getDouble(Object value){
+ return Double.parseDouble(value.toString());
+ }
+
+ public static float getFloat(Object value){
+ return Float.parseFloat(value.toString());
+ }
+
+
+
+ public static Object getRecord(Schema avroSchema, Object value) throws SedonaException {
+ try {
+ if (value == null) {
+ if(Schema.Type.NULL.equals(avroSchema.getType())){
+ return null;
+ }
+ if(!Schema.Type.UNION.equals(avroSchema.getType())){
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()));
+ }
+ }
+ if(value instanceof GenericContainer){
+ return value;
+ }
+
+ if (avroSchema == null) {
+ throw new SedonaException("Null Schema Provided");
+ }
+ switch (avroSchema.getType()) {
+ case RECORD:
+ return getRecord(avroSchema, (Map<String, Object>) value);
+ case ARRAY:
+ return getArray(avroSchema, (Collection<Object>) value);
+ case UNION:
+ return getUnion(avroSchema, value);
+ case INT:
+ return getInteger(value);
+ case LONG:
+ return getLong(value);
+ case FLOAT:
+ return getFloat(value);
+ case DOUBLE:
+ return getDouble(value);
+ default:
+ return value;
+ }
+ }catch (SedonaException e){
+ throw e;
+ } catch (Exception e) {
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()),e);
+ }
+ }
+
+ public static Field getField(String columnName, GeometryType geometryType) throws SedonaException {
+ org.apache.sedona.core.io.avro.schema.Schema schema;
+ switch (geometryType) {
+ case CIRCLE: {
+ schema = CircleSchema.getSchema();
+ break;
+ }
+ case POINT: {
+ schema = CoordinateSchema.getSchema();
+ break;
+ }
+ case RECTANGLE:
+ case POLYGON: {
+ schema = PolygonSchema.getSchema();
+ break;
+ }
+ case LINESTRING: {
+ schema = CoordinateArraySchema.getSchema();
+ break;
+ }
+ default:
+ throw new SedonaException("Geometry Type not Supported");
+ }
+ return new Field(columnName, schema);
+ }
+
+ public static Schema getGeometryRecordSchema(String namespace,
+ String name,
+ Field geometryField,
+ Field userColumn) throws SedonaException {
+ return getGeometryRecordSchema(namespace, name, geometryField, Lists.newArrayList(userColumn));
+ }
+
+ public static Schema getGeometryRecordSchema(String namespace,
+ String name,
+ Field geometryField,
+ List<Field> userColumns) throws SedonaException {
+ List<Field> columns = Lists.newArrayList(geometryField);
+ columns.addAll(userColumns);
+ RecordSchema schema = new RecordSchema(namespace,name, columns);
+ return SchemaUtils.SchemaParser.getSchema(schema.getDataType());
+ }
+
+ public static Map<String, Object> getMapFromCircle(Circle circle) {
+ Coordinate center = circle.getCoordinate();
+ Double radius = circle.getRadius();
+ return ImmutableMap.of(CircleSchema.CENTER, getMapFromCoordinate(center), CircleSchema.RADIUS, radius);
+
+ }
+
+ public static Map<String, Double> getMapFromCoordinate(Coordinate coordinate) {
+ return ImmutableMap.of(CoordinateSchema.X_COORDINATE,
+ coordinate.x,
+ CoordinateSchema.Y_COORDINATE,
+ coordinate.y);
+ }
+
+ public static Collection<Map<String, Double>> getCollectionFromCoordinates(Coordinate[] coordinates) {
+ return Arrays.stream(coordinates).map(AvroUtils::getMapFromCoordinate).collect(Collectors.toList());
+ }
+
+ public static Collection<Map<String, Double>> getCollectionFromLineString(LineString lineString) {
+ return getCollectionFromCoordinates(lineString.getCoordinates());
+ }
+
+ public static Map<String, Double> getMapFromPoint(Point point) {
+ return getMapFromCoordinate(point.getCoordinate());
+ }
+
+ public static Map<String, Object> getMapFromPolygon(Polygon polygon) {
+ return ImmutableMap.of(PolygonSchema.EXTERIOR_RING,
+ getCollectionFromCoordinates(polygon.getExteriorRing().getCoordinates()),
+ PolygonSchema.HOLES,
+ IntStream.range(0, polygon.getNumInteriorRing())
+ .mapToObj(polygon::getInteriorRingN)
+ .map(linearRing -> getCollectionFromCoordinates(linearRing.getCoordinates()))
+ .collect(Collectors.toList()));
+ }
+
+ public static GenericRecord getRecordFromGeometry(Schema schema,
+ String geometryColumnName,
+ String userColumn,
+ Object geometryData,
+ Object userData) throws SedonaException {
+ return (GenericRecord) getRecord(schema,
+ ImmutableMap.of(geometryColumnName, geometryData, userColumn, userData));
+ }
+
+ public static GenericRecord getRecordFromGeometry(Schema schema,
+ String geometryColumnName,
+ Object geometryData,
+ Optional<Map<String, Object>> userData) throws SedonaException {
+ return (GenericRecord) getRecord(schema,
+ (Object) new ImmutableMap.Builder<>().putAll(userData.orElse(Collections.EMPTY_MAP))
+ .put(geometryColumnName, geometryData)
+ .build());
+ }
+
+ public static Schema getSchema(GeometryType geometryType,
+ String geometryColumnName,
+ List<Field> userColumns,
+ String namespace,
+ String name) throws SedonaException {
+ Field geometryField = getField(geometryColumnName, geometryType);
+ return getGeometryRecordSchema(namespace, name, geometryField, userColumns);
+ }
+
+ public static GenericRecord getRecord(Geometry geometry,
+ GeometryType type,
+ String geometryColumnName,
+ Schema avroSchema) throws SedonaException {
+ Object geometryData;
+ switch (type) {
+ case CIRCLE: {
+ geometryData = getMapFromCircle((Circle) geometry);
+ break;
+ }
+ case LINESTRING: {
+ geometryData = getCollectionFromLineString((LineString) geometry);
+ break;
+ }
+ case POINT: {
+ geometryData = getMapFromPoint((Point) geometry);
+ break;
+ }
+ case POLYGON:
Review comment:
Yeah we can support MultiPolygons as well. Will add support for Multipolygons as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] geoHeil commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
geoHeil commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-903317588
Would this PR support multiple geometry columns in the parquet file? But one designated one for calculating the statistics at the row-group level?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r686131960
##########
File path: core/src/main/java/org/apache/sedona/core/io/avro/utils/AvroUtils.java
##########
@@ -0,0 +1,263 @@
+package org.apache.sedona.core.io.avro.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.ImmutableMap;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericArray;
+import org.apache.avro.generic.GenericContainer;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.exceptions.SedonaException;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.geometryObjects.schema.CircleSchema;
+import org.apache.sedona.core.geometryObjects.schema.CoordinateArraySchema;
+import org.apache.sedona.core.geometryObjects.schema.CoordinateSchema;
+import org.apache.sedona.core.geometryObjects.schema.PolygonSchema;
+import org.apache.sedona.core.io.avro.SchemaUtils;
+import org.apache.sedona.core.io.avro.constants.AvroConstants;
+import org.apache.sedona.core.io.avro.schema.Field;
+import org.apache.sedona.core.io.avro.schema.RecordSchema;
+import org.locationtech.jts.geom.*;
+
+import java.util.*;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+public class AvroUtils {
+ public static String getNestedNamespace(String namespace, String... name) {
+
+ return String.join(AvroConstants.DOT, namespace, String.join(AvroConstants.DOT, name));
+ }
+
+
+ public static GenericRecord getRecord(Schema avroSchema, Map<String, Object> value) throws SedonaException {
+ GenericRecord record = new GenericData.Record(avroSchema);
+ for (Schema.Field field : avroSchema.getFields()) {
+ record.put(field.name(), getRecord(field.schema(), value.get(field.name())));
+ }
+ return record;
+ }
+
+ public static GenericArray getArray(Schema avroSchema, Collection<Object> values) throws SedonaException {
+
+ List<Object> vals = Lists.newArrayList();
+ for (Object val : values) {
+ vals.add(getRecord(avroSchema.getElementType(), val));
+ }
+ return new GenericData.Array<>(avroSchema, vals);
+ }
+
+ public static Object getUnion(Schema avroSchema, Object value) throws SedonaException{
+ for(Schema typeSchema:avroSchema.getTypes()){
+ try{
+ return getRecord(typeSchema,value);
+ }catch (SedonaException e){
+
+ }
+ }
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()));
+ }
+
+ public static int getInteger(Object value){
+ return Integer.parseInt(value.toString());
+ }
+
+ public static long getLong(Object value){
+ return Long.parseLong(value.toString());
+ }
+
+ public static double getDouble(Object value){
+ return Double.parseDouble(value.toString());
+ }
+
+ public static float getFloat(Object value){
+ return Float.parseFloat(value.toString());
+ }
+
+
+
+ public static Object getRecord(Schema avroSchema, Object value) throws SedonaException {
+ try {
+ if (value == null) {
+ if(Schema.Type.NULL.equals(avroSchema.getType())){
+ return null;
+ }
+ if(!Schema.Type.UNION.equals(avroSchema.getType())){
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()));
+ }
+ }
+ if(value instanceof GenericContainer){
+ return value;
+ }
+
+ if (avroSchema == null) {
+ throw new SedonaException("Null Schema Provided");
+ }
+ switch (avroSchema.getType()) {
+ case RECORD:
+ return getRecord(avroSchema, (Map<String, Object>) value);
+ case ARRAY:
+ return getArray(avroSchema, (Collection<Object>) value);
+ case UNION:
+ return getUnion(avroSchema, value);
+ case INT:
+ return getInteger(value);
+ case LONG:
+ return getLong(value);
+ case FLOAT:
+ return getFloat(value);
+ case DOUBLE:
+ return getDouble(value);
+ default:
+ return value;
+ }
+ }catch (SedonaException e){
+ throw e;
+ } catch (Exception e) {
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()),e);
+ }
+ }
+
+ public static Field getField(String columnName, GeometryType geometryType) throws SedonaException {
+ org.apache.sedona.core.io.avro.schema.Schema schema;
+ switch (geometryType) {
+ case CIRCLE: {
+ schema = CircleSchema.getSchema();
+ break;
+ }
+ case POINT: {
+ schema = CoordinateSchema.getSchema();
+ break;
+ }
+ case RECTANGLE:
+ case POLYGON: {
+ schema = PolygonSchema.getSchema();
+ break;
+ }
+ case LINESTRING: {
+ schema = CoordinateArraySchema.getSchema();
+ break;
+ }
+ default:
+ throw new SedonaException("Geometry Type not Supported");
+ }
+ return new Field(columnName, schema);
+ }
+
+ public static Schema getGeometryRecordSchema(String namespace,
+ String name,
+ Field geometryField,
+ Field userColumn) throws SedonaException {
+ return getGeometryRecordSchema(namespace, name, geometryField, Lists.newArrayList(userColumn));
+ }
+
+ public static Schema getGeometryRecordSchema(String namespace,
+ String name,
+ Field geometryField,
+ List<Field> userColumns) throws SedonaException {
+ List<Field> columns = Lists.newArrayList(geometryField);
+ columns.addAll(userColumns);
+ RecordSchema schema = new RecordSchema(namespace,name, columns);
+ return SchemaUtils.SchemaParser.getSchema(schema.getDataType());
+ }
+
+ public static Map<String, Object> getMapFromCircle(Circle circle) {
+ Coordinate center = circle.getCoordinate();
+ Double radius = circle.getRadius();
+ return ImmutableMap.of(CircleSchema.CENTER, getMapFromCoordinate(center), CircleSchema.RADIUS, radius);
+
+ }
+
+ public static Map<String, Double> getMapFromCoordinate(Coordinate coordinate) {
+ return ImmutableMap.of(CoordinateSchema.X_COORDINATE,
+ coordinate.x,
+ CoordinateSchema.Y_COORDINATE,
+ coordinate.y);
+ }
+
+ public static Collection<Map<String, Double>> getCollectionFromCoordinates(Coordinate[] coordinates) {
+ return Arrays.stream(coordinates).map(AvroUtils::getMapFromCoordinate).collect(Collectors.toList());
+ }
+
+ public static Collection<Map<String, Double>> getCollectionFromLineString(LineString lineString) {
+ return getCollectionFromCoordinates(lineString.getCoordinates());
+ }
+
+ public static Map<String, Double> getMapFromPoint(Point point) {
+ return getMapFromCoordinate(point.getCoordinate());
+ }
+
+ public static Map<String, Object> getMapFromPolygon(Polygon polygon) {
+ return ImmutableMap.of(PolygonSchema.EXTERIOR_RING,
+ getCollectionFromCoordinates(polygon.getExteriorRing().getCoordinates()),
+ PolygonSchema.HOLES,
+ IntStream.range(0, polygon.getNumInteriorRing())
+ .mapToObj(polygon::getInteriorRingN)
+ .map(linearRing -> getCollectionFromCoordinates(linearRing.getCoordinates()))
+ .collect(Collectors.toList()));
+ }
+
+ public static GenericRecord getRecordFromGeometry(Schema schema,
+ String geometryColumnName,
+ String userColumn,
+ Object geometryData,
+ Object userData) throws SedonaException {
+ return (GenericRecord) getRecord(schema,
+ ImmutableMap.of(geometryColumnName, geometryData, userColumn, userData));
+ }
+
+ public static GenericRecord getRecordFromGeometry(Schema schema,
+ String geometryColumnName,
+ Object geometryData,
+ Optional<Map<String, Object>> userData) throws SedonaException {
+ return (GenericRecord) getRecord(schema,
+ (Object) new ImmutableMap.Builder<>().putAll(userData.orElse(Collections.EMPTY_MAP))
+ .put(geometryColumnName, geometryData)
+ .build());
+ }
+
+ public static Schema getSchema(GeometryType geometryType,
+ String geometryColumnName,
+ List<Field> userColumns,
+ String namespace,
+ String name) throws SedonaException {
+ Field geometryField = getField(geometryColumnName, geometryType);
+ return getGeometryRecordSchema(namespace, name, geometryField, userColumns);
+ }
+
+ public static GenericRecord getRecord(Geometry geometry,
+ GeometryType type,
+ String geometryColumnName,
+ Schema avroSchema) throws SedonaException {
+ Object geometryData;
+ switch (type) {
+ case CIRCLE: {
+ geometryData = getMapFromCircle((Circle) geometry);
+ break;
+ }
+ case LINESTRING: {
+ geometryData = getCollectionFromLineString((LineString) geometry);
+ break;
+ }
+ case POINT: {
+ geometryData = getMapFromPoint((Point) geometry);
+ break;
+ }
+ case POLYGON:
Review comment:
The whole code design is such that one should be able to create complex geometries out of the basic ones and the schema for the parquet files would automatically get generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695177875
##########
File path: core/src/main/java/org/apache/sedona/core/geometryObjects/schema/CoordinateSchema.java
##########
@@ -0,0 +1,39 @@
+package org.apache.sedona.core.geometryObjects.schema;
+
+import com.google.common.collect.Lists;
+import org.apache.sedona.core.io.avro.constants.AvroConstants;
+import org.apache.sedona.core.io.avro.schema.Field;
+import org.apache.sedona.core.io.avro.schema.SimpleSchema;
+import org.apache.sedona.core.io.avro.schema.RecordSchema;
+import org.apache.sedona.core.utils.SedonaUtils;
+
+public class CoordinateSchema extends RecordSchema {
+ public static final String X_COORDINATE = "x";
+ public static final String Y_COORDINATE = "y";
+ public static final String COORDINATE = "coordinate";
+
+ private CoordinateSchema() {
+ super(AvroConstants.SEDONA_NAMESPACE,COORDINATE,
+ Lists.newArrayList(new Field(X_COORDINATE, new SimpleSchema(AvroConstants.PrimitiveDataType.DOUBLE)),
+ new Field(Y_COORDINATE, new SimpleSchema(AvroConstants.PrimitiveDataType.DOUBLE))));
+ }
+
+ private static CoordinateSchema schema;
+
+ public static CoordinateSchema getSchema(){
+ if(SedonaUtils.isNull(schema)){
+ synchronized (CoordinateSchema.class){
+ if(SedonaUtils.isNull(schema)){
+ schema = new CoordinateSchema();
+ }
+ }
+ }
+ return schema;
+ }
+
+// public CoordinateSchema(String name, String namespace) {
Review comment:
Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695179581
##########
File path: core/src/main/java/org/apache/sedona/core/geometryObjects/schema/PolygonSchema.java
##########
@@ -0,0 +1,37 @@
+package org.apache.sedona.core.geometryObjects.schema;
+
+import com.google.common.collect.Lists;
+import org.apache.sedona.core.io.avro.constants.AvroConstants;
+import org.apache.sedona.core.io.avro.schema.*;
+import org.apache.sedona.core.io.avro.utils.AvroUtils;
+import org.apache.sedona.core.utils.SedonaUtils;
+
+/**
+ * Polygon Schema class representing AvroSchema of Polygon Geometry
+ */
+public class PolygonSchema extends RecordSchema {
+ public static final String POLYGON = "polygon";
+ public static final String EXTERIOR_RING = "ex";
+ public static final String HOLES = "holes";
+
+ private PolygonSchema() {
+ super(AvroConstants.SEDONA_NAMESPACE,POLYGON,
+ Lists.newArrayList(new Field(EXTERIOR_RING, CoordinateArraySchema.getSchema()),
+ new Field(HOLES, new UnionSchema(
+ new ArraySchema(CoordinateArraySchema.getSchema()),
+ new SimpleSchema(AvroConstants.PrimitiveDataType.NULL)))));
Review comment:
Saving null would be a better approach to go about I think. Though the default implmentation of the geotools returns an empty array. But I feel we should also support null types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu merged pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu merged pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-903323728
> Would this PR support multiple geometry columns in the parquet file? But one designated one for calculating the statistics at the row-group level?
Currently only one single geometry column per record. We can go ahead and support such features in later versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-897054243
@swamirishi Thanks for your great work. This PR sounds interesting to me. However, this seems to be a big PR so we need to have a thorough review for this.
So far, this PR contains many file changes: 47 files changed so far. Can you avoid changing POM files unless it is necessary? Currently, you duplicate the dependencies to different Sedona modules. Please only keep your changes to Sedona core module if possible. This will be helpful for us to review.
Can you also make your PR pass our CI test?
@Imbruced @netanel246 @Sarwat Folks, please take a look when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695185540
##########
File path: core/src/main/java/org/apache/sedona/core/io/parquet/ParquetFileReader.java
##########
@@ -0,0 +1,60 @@
+package org.apache.sedona.core.io.parquet;
+
+import com.google.common.collect.Lists;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.LongWritable;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.parquet.avro.AvroParquetInputFormat;
+import org.apache.parquet.cli.util.Expressions;
+import org.apache.parquet.cli.util.Schemas;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.sedona.core.constants.SedonaConstants;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import scala.Tuple2;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+public class ParquetFileReader {
+ /**
+ * Reads Parquet File with given geometry Column and the relevant User columns
+ * @param sc
+ * @param geometryColumn
+ * @param userColumns
+ * @param inputPaths
+ * @return Avro Record RDD which needs to be deserialized into a GeometryRDD
+ * @throws IOException
+ */
+ public static JavaRDD<GenericRecord> readFile(JavaSparkContext sc,
+ String geometryColumn,
+ List<String> userColumns,
+ String... inputPaths) throws IOException {
+ final Job job = Job.getInstance(sc.hadoopConfiguration());
Review comment:
> is that mean that current solution works only in hadoop ecosystem ?
This is to import hadoop based configurations if at all there are any. Nothing else but that. Basically getting all the resource paths based configuration. If you look into the code of sc.textFile() it would be doing the same thing internally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r686132465
##########
File path: core/src/main/java/org/apache/sedona/core/io/avro/utils/AvroUtils.java
##########
@@ -0,0 +1,263 @@
+package org.apache.sedona.core.io.avro.utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.ImmutableMap;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericArray;
+import org.apache.avro.generic.GenericContainer;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.exceptions.SedonaException;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.geometryObjects.schema.CircleSchema;
+import org.apache.sedona.core.geometryObjects.schema.CoordinateArraySchema;
+import org.apache.sedona.core.geometryObjects.schema.CoordinateSchema;
+import org.apache.sedona.core.geometryObjects.schema.PolygonSchema;
+import org.apache.sedona.core.io.avro.SchemaUtils;
+import org.apache.sedona.core.io.avro.constants.AvroConstants;
+import org.apache.sedona.core.io.avro.schema.Field;
+import org.apache.sedona.core.io.avro.schema.RecordSchema;
+import org.locationtech.jts.geom.*;
+
+import java.util.*;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+public class AvroUtils {
+ public static String getNestedNamespace(String namespace, String... name) {
+
+ return String.join(AvroConstants.DOT, namespace, String.join(AvroConstants.DOT, name));
+ }
+
+
+ public static GenericRecord getRecord(Schema avroSchema, Map<String, Object> value) throws SedonaException {
+ GenericRecord record = new GenericData.Record(avroSchema);
+ for (Schema.Field field : avroSchema.getFields()) {
+ record.put(field.name(), getRecord(field.schema(), value.get(field.name())));
+ }
+ return record;
+ }
+
+ public static GenericArray getArray(Schema avroSchema, Collection<Object> values) throws SedonaException {
+
+ List<Object> vals = Lists.newArrayList();
+ for (Object val : values) {
+ vals.add(getRecord(avroSchema.getElementType(), val));
+ }
+ return new GenericData.Array<>(avroSchema, vals);
+ }
+
+ public static Object getUnion(Schema avroSchema, Object value) throws SedonaException{
+ for(Schema typeSchema:avroSchema.getTypes()){
+ try{
+ return getRecord(typeSchema,value);
+ }catch (SedonaException e){
+
+ }
+ }
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()));
+ }
+
+ public static int getInteger(Object value){
+ return Integer.parseInt(value.toString());
+ }
+
+ public static long getLong(Object value){
+ return Long.parseLong(value.toString());
+ }
+
+ public static double getDouble(Object value){
+ return Double.parseDouble(value.toString());
+ }
+
+ public static float getFloat(Object value){
+ return Float.parseFloat(value.toString());
+ }
+
+
+
+ public static Object getRecord(Schema avroSchema, Object value) throws SedonaException {
+ try {
+ if (value == null) {
+ if(Schema.Type.NULL.equals(avroSchema.getType())){
+ return null;
+ }
+ if(!Schema.Type.UNION.equals(avroSchema.getType())){
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()));
+ }
+ }
+ if(value instanceof GenericContainer){
+ return value;
+ }
+
+ if (avroSchema == null) {
+ throw new SedonaException("Null Schema Provided");
+ }
+ switch (avroSchema.getType()) {
+ case RECORD:
+ return getRecord(avroSchema, (Map<String, Object>) value);
+ case ARRAY:
+ return getArray(avroSchema, (Collection<Object>) value);
+ case UNION:
+ return getUnion(avroSchema, value);
+ case INT:
+ return getInteger(value);
+ case LONG:
+ return getLong(value);
+ case FLOAT:
+ return getFloat(value);
+ case DOUBLE:
+ return getDouble(value);
+ default:
+ return value;
+ }
+ }catch (SedonaException e){
+ throw e;
+ } catch (Exception e) {
+ throw new SedonaException(String.format("Error while forming Record Given Schema: %s, Given Record: %s",avroSchema.toString(),value==null?null:value.toString()),e);
+ }
+ }
+
+ public static Field getField(String columnName, GeometryType geometryType) throws SedonaException {
+ org.apache.sedona.core.io.avro.schema.Schema schema;
+ switch (geometryType) {
+ case CIRCLE: {
+ schema = CircleSchema.getSchema();
+ break;
+ }
+ case POINT: {
+ schema = CoordinateSchema.getSchema();
+ break;
+ }
+ case RECTANGLE:
+ case POLYGON: {
+ schema = PolygonSchema.getSchema();
+ break;
+ }
+ case LINESTRING: {
+ schema = CoordinateArraySchema.getSchema();
+ break;
+ }
+ default:
+ throw new SedonaException("Geometry Type not Supported");
+ }
+ return new Field(columnName, schema);
+ }
+
+ public static Schema getGeometryRecordSchema(String namespace,
+ String name,
+ Field geometryField,
+ Field userColumn) throws SedonaException {
+ return getGeometryRecordSchema(namespace, name, geometryField, Lists.newArrayList(userColumn));
+ }
+
+ public static Schema getGeometryRecordSchema(String namespace,
+ String name,
+ Field geometryField,
+ List<Field> userColumns) throws SedonaException {
+ List<Field> columns = Lists.newArrayList(geometryField);
+ columns.addAll(userColumns);
+ RecordSchema schema = new RecordSchema(namespace,name, columns);
+ return SchemaUtils.SchemaParser.getSchema(schema.getDataType());
+ }
+
+ public static Map<String, Object> getMapFromCircle(Circle circle) {
+ Coordinate center = circle.getCoordinate();
+ Double radius = circle.getRadius();
+ return ImmutableMap.of(CircleSchema.CENTER, getMapFromCoordinate(center), CircleSchema.RADIUS, radius);
+
+ }
+
+ public static Map<String, Double> getMapFromCoordinate(Coordinate coordinate) {
+ return ImmutableMap.of(CoordinateSchema.X_COORDINATE,
+ coordinate.x,
+ CoordinateSchema.Y_COORDINATE,
+ coordinate.y);
+ }
+
+ public static Collection<Map<String, Double>> getCollectionFromCoordinates(Coordinate[] coordinates) {
+ return Arrays.stream(coordinates).map(AvroUtils::getMapFromCoordinate).collect(Collectors.toList());
+ }
+
+ public static Collection<Map<String, Double>> getCollectionFromLineString(LineString lineString) {
+ return getCollectionFromCoordinates(lineString.getCoordinates());
+ }
+
+ public static Map<String, Double> getMapFromPoint(Point point) {
+ return getMapFromCoordinate(point.getCoordinate());
+ }
+
+ public static Map<String, Object> getMapFromPolygon(Polygon polygon) {
+ return ImmutableMap.of(PolygonSchema.EXTERIOR_RING,
+ getCollectionFromCoordinates(polygon.getExteriorRing().getCoordinates()),
+ PolygonSchema.HOLES,
+ IntStream.range(0, polygon.getNumInteriorRing())
+ .mapToObj(polygon::getInteriorRingN)
+ .map(linearRing -> getCollectionFromCoordinates(linearRing.getCoordinates()))
+ .collect(Collectors.toList()));
+ }
+
+ public static GenericRecord getRecordFromGeometry(Schema schema,
+ String geometryColumnName,
+ String userColumn,
+ Object geometryData,
+ Object userData) throws SedonaException {
+ return (GenericRecord) getRecord(schema,
+ ImmutableMap.of(geometryColumnName, geometryData, userColumn, userData));
+ }
+
+ public static GenericRecord getRecordFromGeometry(Schema schema,
+ String geometryColumnName,
+ Object geometryData,
+ Optional<Map<String, Object>> userData) throws SedonaException {
+ return (GenericRecord) getRecord(schema,
+ (Object) new ImmutableMap.Builder<>().putAll(userData.orElse(Collections.EMPTY_MAP))
+ .put(geometryColumnName, geometryData)
+ .build());
+ }
+
+ public static Schema getSchema(GeometryType geometryType,
+ String geometryColumnName,
+ List<Field> userColumns,
+ String namespace,
+ String name) throws SedonaException {
+ Field geometryField = getField(geometryColumnName, geometryType);
+ return getGeometryRecordSchema(namespace, name, geometryField, userColumns);
+ }
+
+ public static GenericRecord getRecord(Geometry geometry,
+ GeometryType type,
+ String geometryColumnName,
+ Schema avroSchema) throws SedonaException {
+ Object geometryData;
+ switch (type) {
+ case CIRCLE: {
+ geometryData = getMapFromCircle((Circle) geometry);
+ break;
+ }
+ case LINESTRING: {
+ geometryData = getCollectionFromLineString((LineString) geometry);
+ break;
+ }
+ case POINT: {
+ geometryData = getMapFromPoint((Point) geometry);
+ break;
+ }
+ case POLYGON:
Review comment:
Currently I have implemented geometries only for a few of them we can go ahead and implement all of the geometries supported by geotools.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi edited a comment on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi edited a comment on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-903323728
> Would this PR support multiple geometry columns in the parquet file? But one designated one for calculating the statistics at the row-group level?
Currently only one single geometry column per record. We can go ahead and support such features in later versions. The stats are stored on Schemas Column Path. So for multiple Columns the Stats stored would be stored on different paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] geoHeil commented on a change in pull request #536: Sedona 36- Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
geoHeil commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r685799341
##########
File path: core/src/main/java/org/apache/sedona/core/spatialRDD/SpatialRDD.java
##########
@@ -564,7 +588,45 @@ public void saveAsWKT(String outputLocation)
}
}).saveAsTextFile(outputLocation);
}
-
+
+ public void saveAsParquet(JavaSparkContext sc,
+ String geometryColumn,
+ List<Field> userColumns,
+ String outputLocation,
+ String namespace,
+ String name) throws SedonaException, IOException {
+ sc.hadoopConfiguration().setBoolean("parquet.enable.summary-metadata", false);
+ final Job job = Job.getInstance(sc.hadoopConfiguration());
+ GeometryType geometryType = this.getGeometryType();
+ Schema schema = AvroUtils.getSchema(geometryType, geometryColumn, userColumns,namespace,
+ name);
+// ParquetOutputFormat.setWriteSupportClass(job, AvroWriteSupport.class);
+ AvroParquetOutputFormat.setSchema(job, schema);
+ this.rawSpatialRDD.mapPartitionsToPair(new PairFlatMapFunction<Iterator<T>, Void, GenericRecord>() {
+ @Override
+ public Iterator<Tuple2<Void, GenericRecord>> call(Iterator<T> iterator) throws Exception {
+ Schema schema = AvroUtils.getSchema(geometryType, geometryColumn, userColumns, namespace,
+ name);
+ Iterable<T> recordIterable = () -> iterator;
+ try {
+ return StreamSupport.stream(recordIterable.spliterator(), false).map(geometry -> {
+ try {
+ GenericRecord genericRecord =
+ AvroUtils.getRecord(geometry, geometryType, geometryColumn, schema);
Review comment:
Why are we not simply choosing WKB for the serialization of the geometry column and keep the rest as a normal parquet file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-908876891
@swamirishi
Thank you for your contribution!
I will accept this PR into a separate branch: parquet-io. I will need to make some changes to this PR myself such as how to package dependencies.
Meanwhile, you can submit new PRs to the sedona/parquet-io branch to continue your next phase implementation.
Thanks,
Jia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-908876891
@swamirishi
Thank you for your contribution!
I will accept this PR into a separate branch: parquet-io. I will need to make some changes to this PR myself such as how to package dependencies.
Meanwhile, you can submit new PRs to the sedona/parquet-io branch to continue your next phase implementation.
Thanks,
Jia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695186281
##########
File path: pom.xml
##########
@@ -351,6 +429,25 @@
</execution>
</executions>
</plugin>
+<!-- <plugin>-->
Review comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-896125604
> @swamirishi Thanks for your contribution. However, currently, Sedona DataFrame can be saved and loaded as Parquet files. I am not sure if we need this function for RDD. We have the adapter to travel between RDD and DataFrame.
> @jiayuasu It is true we can load parquet files through a dataframe. But this is done through WKB as far as I understand. Parquet stores meta data stats at row group level. We can use those stats for predicate push & spatial Joins etc.
E.g. If I have a file having location data spread across the globe. But while querying I am only interested for data located in India. We can use the row group stats for figuring out only the row groups containing India location data. This would reduce IO costs to a great extent. You can take look at the sample stats above for one of the dummy parquet files created.
> Could you elaborate more about the purpose of this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-898781135
@geoHeil @jiayuasu @Sarwat I have fixed all build errors and also have reverted the pom changes. Please take a look at the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695185869
##########
File path: core/src/main/java/org/apache/sedona/core/showcase/Example.java
##########
@@ -52,99 +52,99 @@
public class Example
implements Serializable
{
-
+
/**
* The sc.
*/
public static JavaSparkContext sc;
-
+
/**
* The geometry factory.
*/
static GeometryFactory geometryFactory;
-
+
/**
* The Point RDD input location.
*/
static String PointRDDInputLocation;
-
+
/**
* The Point RDD offset.
*/
static Integer PointRDDOffset;
-
+
/**
* The Point RDD num partitions.
*/
static Integer PointRDDNumPartitions;
-
+
/**
* The Point RDD splitter.
*/
static FileDataSplitter PointRDDSplitter;
-
+
/**
* The Point RDD index type.
*/
static IndexType PointRDDIndexType;
-
+
Review comment:
Sure let me just remove these changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-897238497
@swamirishi The codebase of SerNetCDF is here: https://github.com/jiayuasu/SerNetCDF which is owned by me. I think you can exclude some libraries in SerNetCDF dependency by using "<exclude></exclude>" tag.
I guess the problem is caused by the dependency in SerNetCDF: https://github.com/jiayuasu/SerNetCDF/blob/master/pom.xml#L46
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] Imbruced commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
Imbruced commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-901222286
This PR is massive, need some time to review it fully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695193899
##########
File path: core/src/test/java/org/apache/sedona/core/io/parquet/ParquetFileReaderTest.java
##########
@@ -0,0 +1,87 @@
+package org.apache.sedona.core.io.parquet;
+
+import com.clearspring.analytics.util.Lists;
+import org.apache.commons.io.FileUtils;
+import org.apache.sedona.core.TestBase;
+import org.apache.sedona.core.enums.FileDataSplitter;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.exceptions.SedonaException;
+import org.apache.sedona.core.formatMapper.ParquetReader;
+import org.apache.sedona.core.formatMapper.WktReader;
+import org.apache.sedona.core.formatMapper.WktReaderTest;
+import org.apache.sedona.core.spatialRDD.PointRDD;
+import org.apache.sedona.core.spatialRDD.SpatialRDD;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.locationtech.jts.geom.Point;
+import org.locationtech.jts.util.Assert;
+
+import java.io.IOException;
+import java.nio.file.FileSystem;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class ParquetFileReaderTest extends TestBase {
+ private static String filePath;
+ @BeforeClass
+ public static void onceExecutedBeforeAll()
+ throws IOException
+ {
+ initialize(ParquetFileReaderTest.class.getName());
+ filePath = ParquetFileReaderTest.class.getClassLoader().getResource("crs-test-point.csv").getPath();
+ }
+
+ @AfterClass
+ public static void tearDown()
+ throws Exception
+ {
+ sc.stop();
+ }
+
+ @Before
+ public void delete() throws IOException {
+ if(testOutputPathDir.exists()){
+ FileUtils.deleteDirectory(testOutputPathDir);
+ }
+ }
+
+ @Test
+ public void testParquetReader() throws IOException, SedonaException {
+ PointRDD pointRDD = new PointRDD(sc, filePath, 0, FileDataSplitter.CSV, false, 1);
+ pointRDD.saveAsParquet(sc,
+ "p",
+ Lists.newArrayList(),
+ testOutputPathDir.getAbsolutePath(),
+ "test.namespace",
+ "name");
+ List<Point> l1 = pointRDD.getRawSpatialRDD().collect().stream().collect(Collectors.toList());
+ Collections.sort(l1, new Comparator<Point>() {
+ @Override
+ public int compare(Point o1, Point o2) {
+ return o1.compareTo(o2);
+ }
+ });
+
+ SpatialRDD<Point> pointRDD1 = ParquetReader.readToGeometryRDD(sc,
+ Arrays.stream(testOutputPathDir.listFiles())
Review comment:
We can, but if you see the text read and write contracts, they are the same. I wanted to keep the contracts as similar as possible excepting for the geometry column and projection column info. But if you feel we should have a contract for getting files based for FileGlobs we can sure do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-901365241
> What about essential documentation updates ? Am I missing them ?
@Imbruced I have pushed the changes containing Documentation Comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-905314053
@swamirishi
Thank you again for your patience! I traveled back to WSU and the new semester just started. So I was kind of overwhelmed by many things on my plate. Now I am ready to work with you!
I personally are very excited about this PR but I think the reason why several committers hesitate to approve this PR is that
(1) it does not have a detailed proposal to explain the proposed file structure and how it works. We know the overall idea of having a geospatial Parquet file and it is an awesome idea for sure. But it is unclear to us how this is achieved in this PR.
(2) it contains 44 file changes which are too many to be reviewed by a human... It touches several critical places in Sedona. For instance, it adds a new module called Parquet, adds some Sedona exceptions, change some dependencies. More importantly, it changes the core/SpatialRDD.java file which will affect many places.
Therefore, I believe here are some action items
(1) would you please add a Sedona website doc in this PR to explain
* The usage of Sedona Parquet reader and writer
* The structure of the proposed geo Parquet file, especially the metadata structure
* the algorithm: how it skips irrelevant data chunks by comparing the spatial query predicate and Parquet metadata
You can put this doc at https://sedona.apache.org/tutorial/ as a programming guide for geospatial Parquet
(2) for the file changes themselves
* please fix the comments raised by @Imbruced
* Please address my comment for SpatialRDD change
* please remove all whitespace file changes, and changes that just include "re-formatting the code"
Once these changes + doc are done and we fully understand what this PR is doing exactly, I will approve this PR. Let the customers give it a try. I believe once this PR is accepted, your follow-up PRs for the Parquet file will quickly pass our review.
Thank you again for your contribution. We really appreciate your contribution!
Jia Yu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi closed pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi closed pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-908097157
> Once these changes + doc are done and we fully understand what this PR is doing exactly, I will approve this PR. Let the customers give it a try. I believe once this PR is accepted, your follow-up PRs for the Parquet file will quickly pass our review.
@jiayuasu @Imbruced Please take a look at the PR now. I have made the required changes for supporting Generic Geometry Type SpatialRDD. I have added a section pertaining to SpatialRDD - parquet Deserialization & Serialization in the file docs/tutorial/rdd.md.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] jiayuasu commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
jiayuasu commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695540349
##########
File path: core/src/main/java/org/apache/sedona/core/spatialRDD/SpatialRDD.java
##########
@@ -129,9 +138,24 @@
* The sample number.
*/
private int sampleNumber = -1;
-
- public int getSampleNumber()
- {
+ /**
+ * Geometry Type Defaults to Geometry Collection
+ */
+ private GeometryType geometryType;
+
+ public SpatialRDD() {
Review comment:
Is there a reason why you need to add geometry type to SpatialRDD? Because you need to use this information to define the type of geospatial objects in a Parquet file?
I don't think this is a good idea. Currently, this generic SpatialRDD is not bounded to a particular type. It allows mixed spatial objects in an RDD. E.g.,
POINT (XXX)
POLYGON (XXX)
MULTIPOLYGON (XXX)
POINT (XXX)
This scenario is actually quite common in real world. A WKT file of state boundaries or country boundaries may consist of both POLYGON and MULTIPOLYGON.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695193899
##########
File path: core/src/test/java/org/apache/sedona/core/io/parquet/ParquetFileReaderTest.java
##########
@@ -0,0 +1,87 @@
+package org.apache.sedona.core.io.parquet;
+
+import com.clearspring.analytics.util.Lists;
+import org.apache.commons.io.FileUtils;
+import org.apache.sedona.core.TestBase;
+import org.apache.sedona.core.enums.FileDataSplitter;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.exceptions.SedonaException;
+import org.apache.sedona.core.formatMapper.ParquetReader;
+import org.apache.sedona.core.formatMapper.WktReader;
+import org.apache.sedona.core.formatMapper.WktReaderTest;
+import org.apache.sedona.core.spatialRDD.PointRDD;
+import org.apache.sedona.core.spatialRDD.SpatialRDD;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.locationtech.jts.geom.Point;
+import org.locationtech.jts.util.Assert;
+
+import java.io.IOException;
+import java.nio.file.FileSystem;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class ParquetFileReaderTest extends TestBase {
+ private static String filePath;
+ @BeforeClass
+ public static void onceExecutedBeforeAll()
+ throws IOException
+ {
+ initialize(ParquetFileReaderTest.class.getName());
+ filePath = ParquetFileReaderTest.class.getClassLoader().getResource("crs-test-point.csv").getPath();
+ }
+
+ @AfterClass
+ public static void tearDown()
+ throws Exception
+ {
+ sc.stop();
+ }
+
+ @Before
+ public void delete() throws IOException {
+ if(testOutputPathDir.exists()){
+ FileUtils.deleteDirectory(testOutputPathDir);
+ }
+ }
+
+ @Test
+ public void testParquetReader() throws IOException, SedonaException {
+ PointRDD pointRDD = new PointRDD(sc, filePath, 0, FileDataSplitter.CSV, false, 1);
+ pointRDD.saveAsParquet(sc,
+ "p",
+ Lists.newArrayList(),
+ testOutputPathDir.getAbsolutePath(),
+ "test.namespace",
+ "name");
+ List<Point> l1 = pointRDD.getRawSpatialRDD().collect().stream().collect(Collectors.toList());
+ Collections.sort(l1, new Comparator<Point>() {
+ @Override
+ public int compare(Point o1, Point o2) {
+ return o1.compareTo(o2);
+ }
+ });
+
+ SpatialRDD<Point> pointRDD1 = ParquetReader.readToGeometryRDD(sc,
+ Arrays.stream(testOutputPathDir.listFiles())
Review comment:
We can, but if you see the text read and write contracts, they are the same. I wanted to keep the contracts as similar as possible excepting for the geometry column and projection column info. But if you feel we should have a contract for getting files based for FileGlobs we can surely do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695176630
##########
File path: core/src/main/java/org/apache/sedona/core/formatMapper/ParquetReader.java
##########
@@ -0,0 +1,38 @@
+package org.apache.sedona.core.formatMapper;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.formatMapper.parquet.ParquetFormatMapper;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.io.parquet.ParquetFileReader;
+import org.apache.sedona.core.spatialRDD.SpatialRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.locationtech.jts.geom.Geometry;
+import org.locationtech.jts.geom.GeometryFactory;
+import org.locationtech.jts.geom.LineString;
+import org.locationtech.jts.geom.Polygon;
+
+import java.io.IOException;
+import java.util.List;
+
+public class ParquetReader extends RddReader {
+ public static <T extends Geometry> SpatialRDD<T> createSpatialRDD(JavaRDD rawRDD,
+ ParquetFormatMapper<T> formatMapper,
+ GeometryType geometryType) {
+ SpatialRDD spatialRDD = new SpatialRDD<T>(geometryType);
+ spatialRDD.rawSpatialRDD = rawRDD.mapPartitions(formatMapper);
+ return spatialRDD;
+ }
+
+ public static <T extends Geometry> SpatialRDD<T> readToGeometryRDD(JavaSparkContext sc,
Review comment:
Currently we need to support reading of random parquet file input as well. We depend on the user to know the data in the parquet file. Otherwise it doesn't really make sense. As a user the user might want to do some projections thus one would need to know the files meta info. Though we can add a feature to identify geometry column in parquet files created by Sedona Framework.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-905344497
@Imbruced I have addressed the review comments. Can you check if I need to change something else
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695177738
##########
File path: core/src/main/java/org/apache/sedona/core/formatMapper/parquet/ParquetFormatMapper.java
##########
@@ -0,0 +1,215 @@
+package org.apache.sedona.core.formatMapper.parquet;
+
+import org.apache.avro.generic.GenericArray;
+import org.apache.avro.generic.GenericContainer;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.log4j.Logger;
+import org.apache.sedona.core.enums.GeometryType;
+import org.apache.sedona.core.formatMapper.FormatMapper;
+import org.apache.sedona.core.geometryObjects.Circle;
+import org.apache.sedona.core.geometryObjects.schema.CircleSchema;
+import org.apache.sedona.core.geometryObjects.schema.CoordinateSchema;
+import org.apache.sedona.core.geometryObjects.schema.PolygonSchema;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import org.apache.zookeeper.Op;
+import org.locationtech.jts.geom.*;
+
+import java.io.Serializable;
+import java.util.*;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import java.util.stream.StreamSupport;
+
+/**
+ * ParquetFormatMapper class
+ * Responsible for converting Avro-Parquet Records into Geometry Objects
+ * @param <T> Geometry Class Type
+ */
+public class ParquetFormatMapper<T extends Geometry> implements Serializable, FlatMapFunction<Iterator<GenericRecord>, T> {
+ final static Logger logger = Logger.getLogger(FormatMapper.class);
+ protected GeometryFactory factory = new GeometryFactory();
+ private GeometryType geometryType;
+ private final String geometryColumn;
+ private final List<String> userColumns;
+
+ /**
+ *
+ * @param geometryType
+ * @param geometryColumn
+ * @param userColumns
+ */
+ public ParquetFormatMapper(GeometryType geometryType, String geometryColumn, List<String> userColumns) {
+ this.geometryType = geometryType;
+ this.geometryColumn = geometryColumn;
+ this.userColumns = userColumns;
+ }
+
+ /**
+ * Gets Coordinate from Avro Record
+ * @param record Avro Record
+ * @return Coordinate
+ */
+ private static Coordinate getCoordinate(GenericRecord record) {
+ Double x = (Double) record.get(CoordinateSchema.X_COORDINATE);
+ Double y = (Double) record.get(CoordinateSchema.Y_COORDINATE);
+ return new Coordinate(x, y);
+ }
+
+ /**
+ * Gets Coordinate from Avro based Record
+ * @param val
+ * @return Coordinate
+ */
+ private static Coordinate getCoordinate(GenericContainer val) {
+ if (val instanceof GenericRecord) {
+ return getCoordinate((GenericRecord) val);
+ }
+ return getCoordinate((GenericArray) val);
+ }
+ /**
+ * Gets Coordinate from Avro based Array based on the first index
+ * @param array
+ * @return Coordinate
+ */
+ private static Coordinate getCoordinate(GenericArray array) {
+ return getCoordinate((GenericRecord) array.get(0));
+ }
+
+ /**
+ * Gets Coordinate Array from Avro Record Array
+ * @param array
+ * @return Coordinate Array
+ */
+ private static Coordinate[] getCoordinates(GenericArray array) {
+ Coordinate[] coordinates = new Coordinate[array.size()];
+ for(int i=0;i<array.size();i++){
+ coordinates[i] = getCoordinate((GenericRecord) array.get(i));
+ }
+ return coordinates;
+ }
+
+ /**
+ * Gets Point Geometry from Coordinate
+ * @param coordinate
+ * @return Point
+ */
+ private Point getPoint(Coordinate coordinate) {
+ return factory.createPoint(coordinate);
+ }
+
+ /**
+ * Gets Circle Geometry Object from Avro Record
+ * @param record
+ * @return Circle
+ */
+ private Circle getCircle(GenericRecord record) {
+ Coordinate center = getCoordinate((GenericRecord) record.get(CircleSchema.CENTER));
+ Double radius = (Double) record.get(CircleSchema.RADIUS);
+ return new Circle(getPoint(center), radius);
+ }
+ /**
+ * Gets Polygon Geometry Object from Avro Record
+ * @param record
+ * @return Polygon
+ */
+ private Polygon getPolygon(GenericRecord record) {
+ LinearRing exteriorRing =
+ factory.createLinearRing(getCoordinates((GenericArray) record.get(PolygonSchema.EXTERIOR_RING)));
+ Optional<GenericArray> holes = Optional.ofNullable((GenericArray) record.get(PolygonSchema.HOLES));
+ List<LinearRing> interiorRings = holes.map(arr->IntStream.range(0, arr.size())
+ .mapToObj(arr::get)
+ .map(hole -> getCoordinates((GenericArray) (hole)))
+ .map(c -> factory.createLinearRing(c))
+ .collect(Collectors.toList())).orElse(Collections.EMPTY_LIST);
+
+ return factory.createPolygon(exteriorRing, (LinearRing[]) interiorRings.toArray(new LinearRing[interiorRings.size()]));
+ }
+
+ /**
+ * Gets Geometry Object of Geometry Type from Coordinate Array
+ * @param coordinates CoordinateArray
+ * @param type geometryType
+ * @return Geometry Object
+ */
+ private Geometry getGeometry(Coordinate[] coordinates, GeometryType type) {
+ Geometry geometry = null;
+ switch (type) {
+ case POINT:
+ geometry = factory.createPoint(coordinates[0]);
+ break;
+ case POLYGON:
+ geometry = factory.createPolygon(coordinates);
Review comment:
This is a defailt implementation wherein a user might just want convert an array of points into a closed polygon without holes. This is similar to supporting Offsets and number of columns length in text format for geometry column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] Imbruced commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
Imbruced commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-901223445
What about essential documentation updates ? Am I missing them ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#issuecomment-897130663
@jiayuasu Is there a code base for org.datasyslab:sernetcdf? I see a bunch of classes google.common classes coming as part of libraries. This has version conflicts with the parquet library due to which I am getting runtime exceptions. In order to do away this I had to shade Parquet libs which isn't possible with the existing pom structure. Dependency management was the way to go about to circumvent the issue without touching the sernetcdf code. If there is a code base for sernetcdf can we look into changing the pom there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695185286
##########
File path: core/src/main/java/org/apache/sedona/core/io/parquet/ParquetFileReader.java
##########
@@ -0,0 +1,60 @@
+package org.apache.sedona.core.io.parquet;
+
+import com.google.common.collect.Lists;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.LongWritable;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.parquet.avro.AvroParquetInputFormat;
+import org.apache.parquet.cli.util.Expressions;
+import org.apache.parquet.cli.util.Schemas;
+import org.apache.parquet.hadoop.ParquetInputFormat;
+import org.apache.sedona.core.constants.SedonaConstants;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.api.java.function.FlatMapFunction;
+import scala.Tuple2;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+public class ParquetFileReader {
+ /**
+ * Reads Parquet File with given geometry Column and the relevant User columns
+ * @param sc
+ * @param geometryColumn
+ * @param userColumns
+ * @param inputPaths
+ * @return Avro Record RDD which needs to be deserialized into a GeometryRDD
+ * @throws IOException
+ */
+ public static JavaRDD<GenericRecord> readFile(JavaSparkContext sc,
+ String geometryColumn,
+ List<String> userColumns,
+ String... inputPaths) throws IOException {
+ final Job job = Job.getInstance(sc.hadoopConfiguration());
Review comment:
This is to import hadoop based configurations if at all there are any. Nothing else but that. Basically getting all the resource paths based configuration. If you look into the code of sc.textFile() it would be doing the same thing internally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sedona] swamirishi commented on a change in pull request #536: [SEDONA-36] Parquet reader & Writers
Posted by GitBox <gi...@apache.org>.
swamirishi commented on a change in pull request #536:
URL: https://github.com/apache/incubator-sedona/pull/536#discussion_r695179581
##########
File path: core/src/main/java/org/apache/sedona/core/geometryObjects/schema/PolygonSchema.java
##########
@@ -0,0 +1,37 @@
+package org.apache.sedona.core.geometryObjects.schema;
+
+import com.google.common.collect.Lists;
+import org.apache.sedona.core.io.avro.constants.AvroConstants;
+import org.apache.sedona.core.io.avro.schema.*;
+import org.apache.sedona.core.io.avro.utils.AvroUtils;
+import org.apache.sedona.core.utils.SedonaUtils;
+
+/**
+ * Polygon Schema class representing AvroSchema of Polygon Geometry
+ */
+public class PolygonSchema extends RecordSchema {
+ public static final String POLYGON = "polygon";
+ public static final String EXTERIOR_RING = "ex";
+ public static final String HOLES = "holes";
+
+ private PolygonSchema() {
+ super(AvroConstants.SEDONA_NAMESPACE,POLYGON,
+ Lists.newArrayList(new Field(EXTERIOR_RING, CoordinateArraySchema.getSchema()),
+ new Field(HOLES, new UnionSchema(
+ new ArraySchema(CoordinateArraySchema.getSchema()),
+ new SimpleSchema(AvroConstants.PrimitiveDataType.NULL)))));
Review comment:
Saving null would be a better approach to go about I think. Though the default implmentation of the geotools returns an empty array. But I feel we should also support null types. I had the same confusion when I was implementing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org