ETL for postgres to hadoop

[Manas Kar <Ma...@exactearth.com>]

Hi All,
                I have some spatial data in postgres machine. I want to be able to move that data to Hadoop and do some geo-processing.
I tried using sqoop to move the data to Hadoop but it complained about the position data(which it says can't recognize)
Does anyone have any idea as to how to do it easily?

Thanks
Manas


<www.exactearth.com>[cid:ee_gradient_tm_150wide.png@f20f7501e5a14d6f85ec33629f725228]<www.exactearth.com>               Manas Kar
Intermediate Software Developer, Product Development | exactEarth Ltd.

60 Struck Ct. Cambridge, Ontario N1R 8L2
office. +1.519.622.4445 ext. 5869 | direct: +1.519.620.5869
email. Manas.Kar@exactearth.com

web. www.exactearth.com






This e-mail and any attachment is for authorized use by the intended recipient(s) only. It contains proprietary or confidential information and is not to be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail, any attachment and all copies and inform the sender. Thank you.

Re: ETL for postgres to hadoop

[andy petrella <an...@gmail.com>]

Hello Manas,

I don't know Sqoop that much but my best guess is that you're probably
using Postgis which has specific structures for Geometry and so on. And if
you need some spatial operators my gut feeling is that things will be
harder ^^ (but a raw import won't need that...).

So I did a quick check in the Sqoop documentation and it looks like
implementing a connector for this kind of structure should do the trick
(check this: http://sqoop.apache.org/docs/1.99.3/ConnectorDevelopment.html).

In any case, I'll be very interested in this kind of stuffs! More than
that, having such import tool for Oracle Spatial cartridge would be great
as well :-P.

my2c,


Andy Petrella
Belgium (Liège)

*       *********
 Data Engineer in *NextLab <http://nextlab.be/> sprl* (owner)
 Engaged Citizen Coder for *WAJUG <http://wajug.be/>* (co-founder)
 Author of *Learning Play! Framework 2
<http://www.packtpub.com/learning-play-framework-2/book>*
 Bio: on visify <https://www.vizify.com/es/52c3feec2163aa0010001eaa>
*       *********
Mobile: *+32 495 99 11 04*
Mails:

   - andy.petrella@nextlab.be
   - andy.petrella@gmail.com

*       *********
Socials:

   - Twitter: https://twitter.com/#!/noootsab
   - LinkedIn: http://be.linkedin.com/in/andypetrella
   - Blogger: http://ska-la.blogspot.com/
   - GitHub:  https://github.com/andypetrella
   - Masterbranch: https://masterbranch.com/andy.petrella



On Tue, Apr 8, 2014 at 10:00 PM, Manas Kar <Ma...@exactearth.com> wrote:

>  Hi All,
>
>                 I have some spatial data in postgres machine. I want to be
> able to move that data to Hadoop and do some geo-processing.
>
> I tried using sqoop to move the data to Hadoop but it complained about the
> position data(which it says can’t recognize)
>
> Does anyone have any idea as to how to do it easily?
>
>
>
> Thanks
>
> Manas
>
>
>
>    <http://www.exactearth.com> <http://www.exactearth.com>   Manas Kar  Intermediate
> Software Developer, Product Development | exactEarth Ltd.     60 Struck
> Ct. Cambridge, Ontario N1R 8L2  office. +1.519.622.4445 ext. 5869 |
> direct: +1.519.620.5869  email. Manas.Kar@exactearth.com
>
> web. www.exactearth.com
>
>
>
>
>  This e-mail and any attachment is for authorized use by the intended
> recipient(s) only. It contains proprietary or confidential information and
> is not to be copied, disclosed to, retained or used by, any other party. If
> you are not an intended recipient then please promptly delete this e-mail,
> any attachment and all copies and inform the sender. Thank you.
>