You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by John <jo...@gmail.com> on 2013/08/21 13:12:19 UTC
Pig Latin Program with special Load Function
Im currently writing a Pig Latin programm:
A = load 'hbase://mytable1' my.packages.CustomizeHBaseStorage('VALUE',
'-loadKey true', 'myrowkey1') as (rowkey:chararray, columncontent:map[]);
ABag = foreach PATTERN_0 generate flatten(my.packages.MapToBag($1)) as
(output:chararray);
the CustimizeHbaseStorage is loading the row "myrowkey1" and after that the
map for this rowkey is transformed to a Bag. That works fine so far.
So, in the ABag are now some entries. With this entries I try to do load
new row keys (every entry in the bag is the information for a new rowkey I
have to load next). So I tried something like this:
X= FOREACH ABag {
TMP = load 'hbase://mytable2'
my.packages.customizeHBaseStorage('VALUE', '-loadKey true', '$0') as
(rowkey:chararray, columncontent:map[]);
GENERATE (TMP.$0);
}
This doesn't does not work, because as far as I now the load statement is
not allowed for FOREACH.
So I tried to build my own EvalFunc:
X = FOREACH INTERMEDIATE_BAG_0 GENERATE my.packages.MyNewUDF($0);
Here is the Java Code for the MyNewUDF:
...
public DataBag exec(Tuple input) throws IOException {
DataBag result = null;
try {
result = bagFactory.newDefaultBag();
CustomizeHBaseStorage loader = new CustomizeHBaseStorage("VALUE",
"-loadKey true", input
.get(0).toString());
loader.getInputFormat();
Tuple curTuple = loader.getNext();
while (curTuple != null) {
result.add(curTuple);
curTuple = loader.getNext();
}
} catch (ParseException e) {
e.printStackTrace();
}
return result;
}
...
I think this would work, but the problem is I got a NullpointerException
because the RecordReader in the HBaseStorage is not initialized when
executing getNext(). So if anybody can say me how I can initialized the
RecordReader (and I think the PigSplit too, because its necessary for
CustomizeHBaseStorage .prepareToRead(RecordReader reader, PigSplit split))
or maybe another approach I would be thankful.
BTW. I know that I can load the whole mytable2 in a new alias and then JOIN
ABag and the new alias, but I try to optimize my program, beacuse it is not
necessary to load the whole mytable2. I try to build a "join" with
information passing.
Thanks
Re: Pig Latin Program with special Load Function
Posted by Pradeep Gollakota <pr...@gmail.com>.
In your eval function, you can use the HBase Get/Scan API to retrieve the
data rather than using the MapReduce API.
On Wed, Aug 21, 2013 at 7:12 AM, John <jo...@gmail.com> wrote:
> Im currently writing a Pig Latin programm:
>
> A = load 'hbase://mytable1' my.packages.CustomizeHBaseStorage('VALUE',
> '-loadKey true', 'myrowkey1') as (rowkey:chararray, columncontent:map[]);
> ABag = foreach PATTERN_0 generate flatten(my.packages.MapToBag($1)) as
> (output:chararray);
>
> the CustimizeHbaseStorage is loading the row "myrowkey1" and after that the
> map for this rowkey is transformed to a Bag. That works fine so far.
>
> So, in the ABag are now some entries. With this entries I try to do load
> new row keys (every entry in the bag is the information for a new rowkey I
> have to load next). So I tried something like this:
>
> X= FOREACH ABag {
> TMP = load 'hbase://mytable2'
> my.packages.customizeHBaseStorage('VALUE', '-loadKey true', '$0') as
> (rowkey:chararray, columncontent:map[]);
> GENERATE (TMP.$0);
> }
>
> This doesn't does not work, because as far as I now the load statement is
> not allowed for FOREACH.
>
> So I tried to build my own EvalFunc:
>
> X = FOREACH INTERMEDIATE_BAG_0 GENERATE my.packages.MyNewUDF($0);
>
> Here is the Java Code for the MyNewUDF:
>
> ...
> public DataBag exec(Tuple input) throws IOException {
> DataBag result = null;
> try {
> result = bagFactory.newDefaultBag();
> CustomizeHBaseStorage loader = new CustomizeHBaseStorage("VALUE",
> "-loadKey true", input
> .get(0).toString());
> loader.getInputFormat();
> Tuple curTuple = loader.getNext();
> while (curTuple != null) {
> result.add(curTuple);
> curTuple = loader.getNext();
> }
> } catch (ParseException e) {
> e.printStackTrace();
> }
> return result;
>
> }
> ...
>
> I think this would work, but the problem is I got a NullpointerException
> because the RecordReader in the HBaseStorage is not initialized when
> executing getNext(). So if anybody can say me how I can initialized the
> RecordReader (and I think the PigSplit too, because its necessary for
> CustomizeHBaseStorage .prepareToRead(RecordReader reader, PigSplit split))
> or maybe another approach I would be thankful.
>
>
> BTW. I know that I can load the whole mytable2 in a new alias and then JOIN
> ABag and the new alias, but I try to optimize my program, beacuse it is not
> necessary to load the whole mytable2. I try to build a "join" with
> information passing.
>
> Thanks
>