You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "Thistlethwaite, Susan - IS" <Su...@exelisinc.com> on 2012/06/26 15:31:26 UTC

Storing Geospatial Tiles in Accumulo Question

ALCON:

My team has been tasked with ingesting/retrieving geospatial tiles to/from Accumulo.

I know others are using Z-Order or Hilbert Curves for storing geospatial points.  Do you know of others who are working with tiles/polygons/shape files?

W/r
Susan Thistlethwaite
Software-Systems Engineer
ITT Exelis
12975 Worldgate Dr.
Herndon, VA 20170
571-332-2662 cell
703-668-6539 office
SusanT@exelisinc.com<ma...@exelisinc.com> email

________________________________
Email addresses of ITT Exelis employees have changed from itt.com to exelisinc.com. Please update your favorites and contact information to reflect these changes.

This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Exelis Inc. The recipient should check this e-mail and any attachments for the presence of viruses. Exelis Inc. accepts no liability for any damage caused by any virus transmitted by this e-mail.

Re: Storing Geospatial Tiles in Accumulo Question

Posted by "Marc P." <ma...@gmail.com>.
If you google "storing hilbert curves," you'll find a few links, which
I used in college to help store this data. it will also prove useful
for you, as you can probably structure and normalize to store them as
keys

On Tue, Jun 26, 2012 at 9:31 AM, Thistlethwaite, Susan - IS
<Su...@exelisinc.com> wrote:
> ALCON:
>
> My team has been tasked with ingesting/retrieving geospatial tiles to/from Accumulo.
>
> I know others are using Z-Order or Hilbert Curves for storing geospatial points.  Do you know of others who are working with tiles/polygons/shape files?
>
> W/r
> Susan Thistlethwaite
> Software-Systems Engineer
> ITT Exelis
> 12975 Worldgate Dr.
> Herndon, VA 20170
> 571-332-2662 cell
> 703-668-6539 office
> SusanT@exelisinc.com<ma...@exelisinc.com> email
>
> ________________________________
> Email addresses of ITT Exelis employees have changed from itt.com to exelisinc.com. Please update your favorites and contact information to reflect these changes.
>
> This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Exelis Inc. The recipient should check this e-mail and any attachments for the presence of viruses. Exelis Inc. accepts no liability for any damage caused by any virus transmitted by this e-mail.

Re: Storing Geospatial Tiles in Accumulo Question

Posted by Ed Kohlwey <ek...@gmail.com>.
David's trick is actually equivalent to a z order curve and a quadtree. Be
aware though that you can't increase magnitude in the portion above the
decimal without throwing the table off, so make sure you pick an
appropriate exponent and stick with it, zero padding out the part above the
decimal.

You can use this to store tiles if you want too, you just use a uniform
point such as the center or upper left hand corner on each tile as the
tile's position. If you address using the corners, you'll also want to use
something like the column family or qualifier to show what depth of the
tesselation you are at because using the corner will result in overlapping
tile addresses. If you need to address the tiles on some other dimension,
you can also use secondary indexing to store references to tiles for some
other purpose.

Sent from my smartphone. Please excuse any typos or shorthand.
On Jun 26, 2012 10:12 PM, "David Arsenault" <da...@gmail.com> wrote:

> Susan
>
> Recently had a discussion about a great geo "trick" for Accumulo. As you
> know Accumulo maintains a sorted set of rows based on RowID, ColFam,
> ColQual. If you had data with LAT/LON values and you interlace the digits
> (fake coods example: lat 203.5, lon 127.1 --> 210237.51), it turns out that
> sorting keeps close points together in the table space. It only works for
> points not tiles, but perhaps you may want some analytics on center points
> of tiles or perhaps you want to efficiently index points of interest for
> structures or other features.
>
> Best,
> David Arsenault
> The MITRE Corporation
>
> Sent from my iPad
>
> On Jun 26, 2012, at 9:31, "Thistlethwaite, Susan - IS" <
> Susan.Thistlethwaite@exelisinc.com> wrote:
>
> > ALCON:
> >
> > My team has been tasked with ingesting/retrieving geospatial tiles
> to/from Accumulo.
> >
> > I know others are using Z-Order or Hilbert Curves for storing geospatial
> points.  Do you know of others who are working with tiles/polygons/shape
> files?
> >
> > W/r
> > Susan Thistlethwaite
> > Software-Systems Engineer
> > ITT Exelis
> > 12975 Worldgate Dr.
> > Herndon, VA 20170
> > 571-332-2662 cell
> > 703-668-6539 office
> > SusanT@exelisinc.com<ma...@exelisinc.com> email
> >
> > ________________________________
> > Email addresses of ITT Exelis employees have changed from itt.com to
> exelisinc.com. Please update your favorites and contact information to
> reflect these changes.
> >
> > This e-mail and any files transmitted with it may be proprietary and are
> intended solely for the use of the individual or entity to whom they are
> addressed. If you have received this e-mail in error please notify the
> sender. Please note that any views or opinions presented in this e-mail are
> solely those of the author and do not necessarily represent those of Exelis
> Inc. The recipient should check this e-mail and any attachments for the
> presence of viruses. Exelis Inc. accepts no liability for any damage caused
> by any virus transmitted by this e-mail.
>

Re: Storing Geospatial Tiles in Accumulo Question

Posted by David Arsenault <da...@gmail.com>.
Susan

Recently had a discussion about a great geo "trick" for Accumulo. As you know Accumulo maintains a sorted set of rows based on RowID, ColFam, ColQual. If you had data with LAT/LON values and you interlace the digits (fake coods example: lat 203.5, lon 127.1 --> 210237.51), it turns out that sorting keeps close points together in the table space. It only works for points not tiles, but perhaps you may want some analytics on center points of tiles or perhaps you want to efficiently index points of interest for structures or other features.

Best,
David Arsenault
The MITRE Corporation

Sent from my iPad

On Jun 26, 2012, at 9:31, "Thistlethwaite, Susan - IS" <Su...@exelisinc.com> wrote:

> ALCON:
> 
> My team has been tasked with ingesting/retrieving geospatial tiles to/from Accumulo.
> 
> I know others are using Z-Order or Hilbert Curves for storing geospatial points.  Do you know of others who are working with tiles/polygons/shape files?
> 
> W/r
> Susan Thistlethwaite
> Software-Systems Engineer
> ITT Exelis
> 12975 Worldgate Dr.
> Herndon, VA 20170
> 571-332-2662 cell
> 703-668-6539 office
> SusanT@exelisinc.com<ma...@exelisinc.com> email
> 
> ________________________________
> Email addresses of ITT Exelis employees have changed from itt.com to exelisinc.com. Please update your favorites and contact information to reflect these changes.
> 
> This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Exelis Inc. The recipient should check this e-mail and any attachments for the presence of viruses. Exelis Inc. accepts no liability for any damage caused by any virus transmitted by this e-mail.

Re: Storing Geospatial Tiles in Accumulo Question

Posted by Ed Kohlwey <ek...@gmail.com>.
You could also use a quadtree, the implementation of which you will find
very similar to a z-order curve.

On Tue, Jun 26, 2012 at 10:13 AM, Adam Fuchs <af...@apache.org> wrote:

> Susan,
>
> As a starting point, check out this article on Geo Hashing:
> http://en.wikipedia.org/wiki/Geohash.
>
> Cheers,
> Adam
>
>
> On Tue, Jun 26, 2012 at 9:31 AM, Thistlethwaite, Susan - IS <
> Susan.Thistlethwaite@exelisinc.com> wrote:
>
> > ALCON:
> >
> > My team has been tasked with ingesting/retrieving geospatial tiles
> to/from
> > Accumulo.
> >
> > I know others are using Z-Order or Hilbert Curves for storing geospatial
> > points.  Do you know of others who are working with tiles/polygons/shape
> > files?
> >
> > W/r
> > Susan Thistlethwaite
> > Software-Systems Engineer
> > ITT Exelis
> > 12975 Worldgate Dr.
> > Herndon, VA 20170
> > 571-332-2662 cell
> > 703-668-6539 office
> > SusanT@exelisinc.com<ma...@exelisinc.com> email
> >
> > ________________________________
> > Email addresses of ITT Exelis employees have changed from itt.com to
> > exelisinc.com. Please update your favorites and contact information to
> > reflect these changes.
> >
> > This e-mail and any files transmitted with it may be proprietary and are
> > intended solely for the use of the individual or entity to whom they are
> > addressed. If you have received this e-mail in error please notify the
> > sender. Please note that any views or opinions presented in this e-mail
> are
> > solely those of the author and do not necessarily represent those of
> Exelis
> > Inc. The recipient should check this e-mail and any attachments for the
> > presence of viruses. Exelis Inc. accepts no liability for any damage
> caused
> > by any virus transmitted by this e-mail.
> >
>

Re: Storing Geospatial Tiles in Accumulo Question

Posted by Adam Fuchs <af...@apache.org>.
Susan,

As a starting point, check out this article on Geo Hashing:
http://en.wikipedia.org/wiki/Geohash.

Cheers,
Adam


On Tue, Jun 26, 2012 at 9:31 AM, Thistlethwaite, Susan - IS <
Susan.Thistlethwaite@exelisinc.com> wrote:

> ALCON:
>
> My team has been tasked with ingesting/retrieving geospatial tiles to/from
> Accumulo.
>
> I know others are using Z-Order or Hilbert Curves for storing geospatial
> points.  Do you know of others who are working with tiles/polygons/shape
> files?
>
> W/r
> Susan Thistlethwaite
> Software-Systems Engineer
> ITT Exelis
> 12975 Worldgate Dr.
> Herndon, VA 20170
> 571-332-2662 cell
> 703-668-6539 office
> SusanT@exelisinc.com<ma...@exelisinc.com> email
>
> ________________________________
> Email addresses of ITT Exelis employees have changed from itt.com to
> exelisinc.com. Please update your favorites and contact information to
> reflect these changes.
>
> This e-mail and any files transmitted with it may be proprietary and are
> intended solely for the use of the individual or entity to whom they are
> addressed. If you have received this e-mail in error please notify the
> sender. Please note that any views or opinions presented in this e-mail are
> solely those of the author and do not necessarily represent those of Exelis
> Inc. The recipient should check this e-mail and any attachments for the
> presence of viruses. Exelis Inc. accepts no liability for any damage caused
> by any virus transmitted by this e-mail.
>