Contributing Apache Sedona(Raster Dataframes)

[Shantanu Aggarwal <sh...@gmail.com>]

Hello All,

I am a current graduate student at Arizona State University and wanted to
propose raster data frames written in Pyspark that can be incorporated in
Apache Sedona to load satellite images and be able to perform various map
algebra operations on it.

How can I add my constructors as a part of the Python folder? Is there a
separate guide on how to contribute?

Hope to hear from you soon!


Very Respectfully
Shantanu Aggarwal
Masters In Science
Arizona State University

Re: Contributing Apache Sedona(Raster Dataframes)

[Jia Yu <ji...@apache.org>]

Hi Shantanu,

First, thanks for your contribution. If your code is implemented in Python,
unfortunately, Sedona Python folder currently only has a number of wrappers
around our Java JVM objects. The design goal is to make sure that once we
implement new APIs in Scala/Java, Sedona Python will be able to easily port
these APIs.

To move forward, I would suggest that

1. Sedona Binder Example Notebooks. If your implementation is purely in
Python or JupyterNotebook, you can make a PR to put it in our Binder folder
along with a JupyterNotebook example (see
https://github.com/apache/incubator-sedona/tree/master/binder). Please try
to keep your core Python functions in .py files such that others can easily
download and import them into their notebooks. The user can also enjoy this
func immediately by clicking our "Launch Binder" icon.

2. After Step 1, you can try to create a folder called "raster" in Sedona
Python and move your .py files there. Then, make another PR. Note that,
this will require detailed test cases for different scenarios such as
file:// and hdfs:// path prefix.

Because Step 2 will take some time to finish and our Scala or Java API
won't be able to share this functionality, the better choice will be
implementing these functions in Sedona SQL (written in Scala, you may want
to use GeoTools for reading rasters locally). But it is up to you depending
on how much time you have.

Thank you again for the contribution!

Jia


On Mon, Apr 19, 2021 at 9:45 PM Mo Sarwat <th...@gmail.com> wrote:

> Shantanu,
>
> It will be a great idea to include GeoTiff capability into Apache Sedona.
> As for the other libraries mentioned by Jim. In case they plan on pushing
> their code as part of the Apache Sedona source code, I believe it will not
> be a good idea for you to push your code. However, in case the developers
> of these libraries work separately from Apache Sedona and do not refer to
> it, I do not see why you cannot propose your own solution based upon the
> Sedona ecosystem. That will benefit the entire Sedona community since they
> do not need to load extra libraries in order to handle GeoTiff data into
> Sedona.
>
> Thanks again for your contribution.
>
> On Mon, Apr 19, 2021 at 10:23 AM Fitch, Simeon <fi...@astraea.io> wrote:
>
> > Shantanu,
> >
> > As Jim alludes to, we'd definitely be interested in collaborating over
> this
> > capability rather than creating competing solutions. There are more
> > technical challenges to solving this problem than people to solve them!
> > RasterFrames <https://rasterframes.io/> already has map algebra ops (an
> > many other) available as > 200 PySpark functions
> > <https://rasterframes.io/reference.html>. Would definitely be interested
> > in
> > getting your opinion on what you think is missing, and where you could
> > contribute!
> >
> > Best,
> >
> > Simeon
> >
> >
> > On Mon, Apr 19, 2021 at 11:01 AM Jim Hughes <jh...@ccri.com> wrote:
> >
> > > Hi Shantanu,
> > >
> > > I'd be interested to know how your work would compare to existing
> > > projects which provide raster support in Spark.  LocationTech
> GeoTrellis
> > > has existed for several years and provides that support already.  Also,
> > > LocationTech RasterFrames builds on top of GeoTrellis to provide
> PySpark
> > > and Spark SQL support for data science with respect to raster-based
> > > dataframes.
> > >
> > > Cheers,
> > >
> > > Jim
> > >
> > > On 4/18/21 1:25 PM, Shantanu Aggarwal wrote:
> > > > Hello All,
> > > >
> > > > I am a current graduate student at Arizona State University and
> wanted
> > to
> > > > propose raster data frames written in Pyspark that can be
> incorporated
> > in
> > > > Apache Sedona to load satellite images and be able to perform various
> > map
> > > > algebra operations on it.
> > > >
> > > > How can I add my constructors as a part of the Python folder? Is
> there
> > a
> > > > separate guide on how to contribute?
> > > >
> > > > Hope to hear from you soon!
> > > >
> > > >
> > > > Very Respectfully
> > > > Shantanu Aggarwal
> > > > Masters In Science
> > > > Arizona State University
> > > >
> > >
> > >
> >
> > --
> > Simeon Fitch
> > Co-founder & VP of R&D
> > Astraea, Inc.
> >
>

Re: Contributing Apache Sedona(Raster Dataframes)

[Mo Sarwat <th...@gmail.com>]

Shantanu,

It will be a great idea to include GeoTiff capability into Apache Sedona.
As for the other libraries mentioned by Jim. In case they plan on pushing
their code as part of the Apache Sedona source code, I believe it will not
be a good idea for you to push your code. However, in case the developers
of these libraries work separately from Apache Sedona and do not refer to
it, I do not see why you cannot propose your own solution based upon the
Sedona ecosystem. That will benefit the entire Sedona community since they
do not need to load extra libraries in order to handle GeoTiff data into
Sedona.

Thanks again for your contribution.

On Mon, Apr 19, 2021 at 10:23 AM Fitch, Simeon <fi...@astraea.io> wrote:

> Shantanu,
>
> As Jim alludes to, we'd definitely be interested in collaborating over this
> capability rather than creating competing solutions. There are more
> technical challenges to solving this problem than people to solve them!
> RasterFrames <https://rasterframes.io/> already has map algebra ops (an
> many other) available as > 200 PySpark functions
> <https://rasterframes.io/reference.html>. Would definitely be interested
> in
> getting your opinion on what you think is missing, and where you could
> contribute!
>
> Best,
>
> Simeon
>
>
> On Mon, Apr 19, 2021 at 11:01 AM Jim Hughes <jh...@ccri.com> wrote:
>
> > Hi Shantanu,
> >
> > I'd be interested to know how your work would compare to existing
> > projects which provide raster support in Spark.  LocationTech GeoTrellis
> > has existed for several years and provides that support already.  Also,
> > LocationTech RasterFrames builds on top of GeoTrellis to provide PySpark
> > and Spark SQL support for data science with respect to raster-based
> > dataframes.
> >
> > Cheers,
> >
> > Jim
> >
> > On 4/18/21 1:25 PM, Shantanu Aggarwal wrote:
> > > Hello All,
> > >
> > > I am a current graduate student at Arizona State University and wanted
> to
> > > propose raster data frames written in Pyspark that can be incorporated
> in
> > > Apache Sedona to load satellite images and be able to perform various
> map
> > > algebra operations on it.
> > >
> > > How can I add my constructors as a part of the Python folder? Is there
> a
> > > separate guide on how to contribute?
> > >
> > > Hope to hear from you soon!
> > >
> > >
> > > Very Respectfully
> > > Shantanu Aggarwal
> > > Masters In Science
> > > Arizona State University
> > >
> >
> >
>
> --
> Simeon Fitch
> Co-founder & VP of R&D
> Astraea, Inc.
>

Re: Contributing Apache Sedona(Raster Dataframes)

["Fitch, Simeon" <fi...@astraea.io>]

Shantanu,

As Jim alludes to, we'd definitely be interested in collaborating over this
capability rather than creating competing solutions. There are more
technical challenges to solving this problem than people to solve them!
RasterFrames <https://rasterframes.io/> already has map algebra ops (an
many other) available as > 200 PySpark functions
<https://rasterframes.io/reference.html>. Would definitely be interested in
getting your opinion on what you think is missing, and where you could
contribute!

Best,

Simeon


On Mon, Apr 19, 2021 at 11:01 AM Jim Hughes <jh...@ccri.com> wrote:

> Hi Shantanu,
>
> I'd be interested to know how your work would compare to existing
> projects which provide raster support in Spark.  LocationTech GeoTrellis
> has existed for several years and provides that support already.  Also,
> LocationTech RasterFrames builds on top of GeoTrellis to provide PySpark
> and Spark SQL support for data science with respect to raster-based
> dataframes.
>
> Cheers,
>
> Jim
>
> On 4/18/21 1:25 PM, Shantanu Aggarwal wrote:
> > Hello All,
> >
> > I am a current graduate student at Arizona State University and wanted to
> > propose raster data frames written in Pyspark that can be incorporated in
> > Apache Sedona to load satellite images and be able to perform various map
> > algebra operations on it.
> >
> > How can I add my constructors as a part of the Python folder? Is there a
> > separate guide on how to contribute?
> >
> > Hope to hear from you soon!
> >
> >
> > Very Respectfully
> > Shantanu Aggarwal
> > Masters In Science
> > Arizona State University
> >
>
>

-- 
Simeon Fitch
Co-founder & VP of R&D
Astraea, Inc.

Re: Contributing Apache Sedona(Raster Dataframes)

[Jim Hughes <jh...@ccri.com>]

Hi Shantanu,

I'd be interested to know how your work would compare to existing 
projects which provide raster support in Spark.  LocationTech GeoTrellis 
has existed for several years and provides that support already.  Also, 
LocationTech RasterFrames builds on top of GeoTrellis to provide PySpark 
and Spark SQL support for data science with respect to raster-based 
dataframes.

Cheers,

Jim

On 4/18/21 1:25 PM, Shantanu Aggarwal wrote:
> Hello All,
>
> I am a current graduate student at Arizona State University and wanted to
> propose raster data frames written in Pyspark that can be incorporated in
> Apache Sedona to load satellite images and be able to perform various map
> algebra operations on it.
>
> How can I add my constructors as a part of the Python folder? Is there a
> separate guide on how to contribute?
>
> Hope to hear from you soon!
>
>
> Very Respectfully
> Shantanu Aggarwal
> Masters In Science
> Arizona State University
>