Problem with Pig Store command

[Alex WANG <AW...@mdacorporation.com>]

Hi,

I am using pig 0.7.0 in hadoop mapreduce mode.

The problem I have is that I simply can't use

STORE INTO alias USING PigStorage();

I can load dataset in, write UDFs to manipulate the dataset, but I can't store it. The output is a directory in HDFS with 0 bytes.

As an example, I've been testing with a simple script:

W = load 'wordbag' using PigStorage(' ') as (f1:int, f2:int, name:chararray, type:chararray);
store W into 'wordtesting' using PigStorage(' ');

I run the code in grunt, and the output of hadoop fs -ls is:

drwxr-xr-x   - awang supergroup          0 2010-09-21 13:45 /user/awang/wordtesting

The grunt messages are:

grunt> store filteredW into 'wordtesting' using PigStorage(' ');
2010-09-21 13:45:35,210 [main] INFO  org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned for W
2010-09-21 13:45:35,210 [main] INFO  org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned for W
2010-09-21 13:45:35,440 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: Store(hdfs://pineal:9000/user/awang/wordtesting:PigStorage(' ')) - 1-46 Operator Key: 1-46)
2010-09-21 13:45:35,498 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2010-09-21 13:45:35,498 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2010-09-21 13:45:35,549 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-09-21 13:45:38,100 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2010-09-21 13:45:38,166 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2010-09-21 13:45:38,173 [Thread-15] WARN  org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2010-09-21 13:45:38,307 [Thread-15] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2010-09-21 13:45:38,307 [Thread-15] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2010-09-21 13:45:38,670 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201009211320_0002
2010-09-21 13:45:38,670 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://pineal:50030/jobdetails.jsp?jobid=job_201009211320_0002
2010-09-21 13:45:38,673 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2010-09-21 13:45:48,755 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2010-09-21 13:45:53,835 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2010-09-21 13:45:53,835 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Successfully stored result in: "hdfs://pineal:9000/user/awang/wordtesting"
2010-09-21 13:45:53,846 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Records written : 1
2010-09-21 13:45:53,846 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Bytes written : 20
2010-09-21 13:45:53,846 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Spillable Memory Manager spill count : 0
2010-09-21 13:45:53,847 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Proactive spill count : 0
2010-09-21 13:45:53,847 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!


I've been struggling with this for a long time.... It works if I have a one bytearray in my tuple, but once I defined my schema, it  no longer works.

Anyone has any idea? Please help!! Thanks!

Best regards,
Alex