Re: Gora + Pig + Cascading + Hadoop Submit Europe 2014

[Alfonso Nishikawa <al...@gmail.com>]

Hi!

Sorry for the delay (I am quite busy at the university -as student-).

I write to comment the content I would show.

I must clarify, first, that at this moment Pig-Gora connector (GoraStorage)
loads full Persistent's fields, and I have to improve it to load only a
subset of the records.
I must clarify too that Cascading-Gora connector (GoraTap) is under
development, and facing the fact that cascading works with the "old" Hadoop
api. For that I am using some classes of elephant-bird.

The contents to show would be:

- About Gora: general info
- Working with Gora: Connect to a datastore and read
- Gora Architecture: Insides
- Pig:
  * Correspondence between Pig and Gora data abstractions. Today mapping is
the following:

null <-> null
boolean <-> boolean
int <-> int
long <-> long
float <-> float
double <-> double
bytes <-> bytearray
string <-> chararray
record <-> tuple
enum <-> int
map <-> map<String,X>
array <-> bag
union <-> <X>
fixed <-> -unsupported-

  * Added restrictions. For example, related to unions it is only allowed
"optional fields" (['null','type']).
  * Loading and saving in Pig (full and partial record's fields, commented
at top of this message).
  * Some format details of the data loaded with Gora.

- Cascading:
  * Tap and Scheme: loading and saving.
  * Loading/saving the Persistent instance as a Tuple(Persistent) -first
development approach-
  * Functions to convert Tuple(Persistent) to Tuple(field,field,field,...)
and viceversa, and the problems and possible solutions for managing the
state of a Persistent instance when created form
Tuple(field,field,field,...). At this point I don't expect to have enough
time to implementing this, but we will have a shell for further
developments. My needs by now (time/cost) does not reach this point (but at
long term, of course).

- Benefits using Gora
- Deawbacks using Gora


I accept suggestions, of course :) (but I actually don't expect to be
selected for Hadoop Submit :P)

Alfonso





2013/11/28 Lewis John Mcgibbney <le...@gmail.com>

> Hey Alfonso,
> Thanks for the update
>
> On Thu, Nov 28, 2013 at 3:26 PM, <de...@gora.apache.org> wrote:
>
> >
> > Thanks for the support :)
> >
>
> Right back at ya
>
>
> >
> > Hadoop Submmit 2014 Europe. Track vote is open:
> > https://hadoopsummit.uservoice.com/
> > Gora is in "Committers":
> > https://hadoopsummit.uservoice.com/forums/232059-committers
> >
> >
> > There are plenty of good looking proposals but I backed this with all 3
> of
> my votes. Exposure like this is exactly what we need to grow the community.
> Best of luck Alfonso, please keep us updated.
> Lewis
>