You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Raghu Angadi (JIRA)" <ji...@apache.org> on 2011/07/29 03:56:10 UTC
[jira] [Updated] (PIG-2193) Problem with HBase loader 0.90.3 and
PIG 0.8.1
[ https://issues.apache.org/jira/browse/PIG-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated PIG-2193:
------------------------------
Attachment: PIG-2193.patch
The patch ignores the scanner conf in jobConf. This seems to fix my script.
> Problem with HBase loader 0.90.3 and PIG 0.8.1
> ----------------------------------------------
>
> Key: PIG-2193
> URL: https://issues.apache.org/jira/browse/PIG-2193
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.1
> Environment: HBase 0.90.3, Hadoop 0.20-append
> Reporter: Vincent BARAT
> Attachments: PIG-2193.patch
>
>
> I've some data in HBase 0.90.3 and I run a simple script on them.
> This script badly returns 0 records. From time to time, under yet undefined conditions, the same script on the same data works (it return correct data).
> When data are loaded from HDFS instead of HBase, the script runs perfectly.
> Here is the script loading from HDFS (works):
> start_sessions = LOAD 'start_sessions' AS (sid:chararray, infoid:chararray, imei:chararray, start:long);
> end_sessions = LOAD 'end_sessions' AS (sid:chararray, end:long, locid:chararray);
> infos = LOAD 'infos' AS (infoid:chararray, network_type:chararray, network_subtype:chararray, locale:chararray, version_name:chararray, carrier_country:chararray, carrier_name:chararray, phone_manufacturer:chararray, phone_model:chararray, firmware_version:chararray, firmware_name:chararray);
> sessions = JOIN start_sessions BY sid, end_sessions BY sid;
> sessions = FILTER sessions BY end > start AND end - start < 86400000L;
> sessions = JOIN sessions BY infoid, infos BY infoid;
> sessions = LIMIT sessions 100;
> dump sessions;
> The same script loading from HBase (don't work):
> start_sessions = LOAD 'startSession' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid meta:infoid meta:imei meta:timestamp') AS (sid:chararray, infoid:chararray, imei:chararray, start:long);
> end_sessions = LOAD 'endSession' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid meta:timestamp meta:locid') AS (sid:chararray, end:long, locid:chararray);
> infos = LOAD 'info' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:infoid data:networkType data:networkSubtype data:locale data:applicationVersionName data:carrierCountry data:carrierName data:phoneManufacturer data:phoneModel data:firmwareVersion data:firmwareName') AS (infoid:chararray, network_type:chararray, network_subtype:chararray, locale:chararray, version_name:chararray, carrier_country:chararray, carrier_name:chararray, phone_manufacturer:chararray, phone_model:chararray, firmware_version:chararray, firmware_name:chararray);
> sessions = JOIN start_sessions BY sid, end_sessions BY sid;
> sessions = FILTER sessions BY end > start AND end - start < 86400000L;
> sessions = JOIN sessions BY infoid, infos BY infoid;
> sessions = LIMIT sessions 100;
> dump sessions;
> I guess it definitively means there is a nasty bug in the HBase loader.
> Here is the PIG dump for the HBase version:
> aws09:~# pig
> 2011-07-28 08:17:36,329 [main] INFO org.apache.pig.Main - Logging error messages to: /root/pig_1311841056328.log
> 2011-07-28 08:17:36,641 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://aws09.preprod.ubithere.com:9000
> 2011-07-28 08:17:36,923 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: aws09.preprod.ubithere.com:9001
> grunt> start_sessions = LOAD 'startSession.mde253811.preprod.ubithere.com' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid meta:infoid meta:imei meta:timestamp') AS (sid:chararray, infoid:chararray, imei:chararray, start:long);
> grunt> end_sessions = LOAD 'endSession.mde253811.preprod.ubithere.com' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:sid meta:timestamp meta:locid') AS (sid:chararray, end:long, locid:chararray);
> grunt> infos = LOAD 'info.mde253811.preprod.ubithere.com' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('meta:infoid data:networkType data:networkSubtype data:locale data:applicationVersionName data:carrierCountry data:carrierName data:phoneManufacturer data:phoneModel data:firmwareVersion data:firmwareName') AS (infoid:chararray, network_type:chararray, network_subtype:chararray, locale:chararray, version_name:chararray, carrier_country:chararray, carrier_name:chararray, phone_manufacturer:chararray, phone_model:chararray, firmware_version:chararray, firmware_name:chararray);
> grunt> sessions = JOIN start_sessions BY sid, end_sessions BY sid;
> grunt> sessions = FILTER sessions BY end > start AND end - start < 86400000L;
> grunt> sessions = JOIN sessions BY infoid, infos BY infoid;
> grunt> sessions = LIMIT sessions 100;
> grunt> dump sessions;
> 2011-07-28 08:17:50,275 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: HASH_JOIN,FILTER,LIMIT
> 2011-07-28 08:17:50,275 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-07-28 08:17:51,213 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: sessions: Store(hdfs://aws09.preprod.ubithere.com:9000/tmp/temp-1404953096/tmp819396740:org.apache.pig.impl.io.InterStorage) - scope-93 Operator Key: scope-93)
> 2011-07-28 08:17:51,225 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2011-07-28 08:17:51,281 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage->POForEach to POJoinPackage
> 2011-07-28 08:17:51,281 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer - Rewrite: POPackage->POForEach to POJoinPackage
> 2011-07-28 08:17:51,350 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 3
> 2011-07-28 08:17:51,350 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 3
> 2011-07-28 08:17:51,402 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
> 2011-07-28 08:17:51,411 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.3.2-1031432, built on 11/05/2010 05:32 GMT
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=aws09.machine.com
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.6.0_22
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Sun Microsystems Inc.
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.22/jre
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/pig/bin/../conf:/usr/lib/jvm/java-6-sun/jre/lib/tools.jar:/opt/pig/bin/../pig-0.8.1-core.jar:/opt/pig/bin/../build/pig-*-SNAPSHOT.jar:/opt/pig/bin/../lib/commons-el-1.0.jar:/opt/pig/bin/../lib/commons-lang-2.4.jar:/opt/pig/bin/../lib/commons-logging-1.1.1.jar:/opt/pig/bin/../lib/guava-r06.jar:/opt/pig/bin/../lib/hbase-0.90.3.jar:/opt/pig/bin/../lib/hsqldb-1.8.0.10.jar:/opt/pig/bin/../lib/jackson-core-asl-1.0.1.jar:/opt/pig/bin/../lib/jackson-mapper-asl-1.0.1.jar:/opt/pig/bin/../lib/javacc-4.2.jar:/opt/pig/bin/../lib/javacc.jar:/opt/pig/bin/../lib/jetty-util-6.1.14.jar:/opt/pig/bin/../lib/jline-0.9.94.jar:/opt/pig/bin/../lib/joda-time-1.6.jar:/opt/pig/bin/../lib/jsch-0.1.38.jar:/opt/pig/bin/../lib/junit-4.5.jar:/opt/pig/bin/../lib/jython-2.5.0.jar:/opt/pig/bin/../lib/log4j-1.2.14.jar:/opt/pig/bin/../lib/pigudfs.jar:/opt/pig/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/pig/bin/../lib/zookeeper-3.3.2.jar:/opt/hadoop/conf_computation:/opt/hbase/conf:/opt/pig/lib/hadoop-0.20-append-core.jar
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.22/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=2.6.21.7-2.fc8xen-ec2-v1.0
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
> 2011-07-28 08:17:51,470 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/root
> 2011-07-28 08:17:51,471 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=aws09.machine.com:2222 sessionTimeout=60000 watcher=hconnection
> 2011-07-28 08:17:51,493 [main-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server aws09.machine.com/10.83.1.244:2222
> 2011-07-28 08:17:51,499 [main-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to aws09.machine.com/10.83.1.244:2222, initiating session
> 2011-07-28 08:17:51,508 [main-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server aws09.machine.com/10.83.1.244:2222, sessionid = 0x131617dada6054b, negotiated timeout = 60000
> 2011-07-28 08:17:51,575 [main] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ef894ce; hsa=aws03.machine.com:60020
> 2011-07-28 08:17:51,687 [main] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for .META.,,1.1028785192 is aws03.machine.com:60020
> 2011-07-28 08:17:51,696 [main] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=endSession.mde253811.preprod.ubithere.com,,00000000000000 for max=10 rows
> 2011-07-28 08:17:51,700 [main] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for endSession.mde253811.preprod.ubithere.com,,1311086199483.706685579 is aws03.machine.com:60020
> 2011-07-28 08:17:51,726 [main] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=startSession.mde253811.preprod.ubithere.com,,00000000000000 for max=10 rows
> 2011-07-28 08:17:51,729 [main] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for startSession.mde253811.preprod.ubithere.com,,1311086198252.1334391323 is aws03.machine.com:60020
> 2011-07-28 08:17:53,328 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
> 2011-07-28 08:17:53,335 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=0
> 2011-07-28 08:17:53,335 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
> 2011-07-28 08:17:53,442 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
> 2011-07-28 08:17:53,944 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
> 2011-07-28 08:17:53,989 [Thread-13] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=aws09.machine.com:2222 sessionTimeout=60000 watcher=hconnection
> 2011-07-28 08:17:53,990 [Thread-13-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server aws09.machine.com/10.83.1.244:2222
> 2011-07-28 08:17:53,991 [Thread-13-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to aws09.machine.com/10.83.1.244:2222, initiating session
> 2011-07-28 08:17:53,996 [Thread-13-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server aws09.machine.com/10.83.1.244:2222, sessionid = 0x131617dada6054c, negotiated timeout = 60000
> 2011-07-28 08:17:54,000 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2d9f90e3; hsa=aws03.machine.com:60020
> 2011-07-28 08:17:54,005 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for .META.,,1.1028785192 is aws03.machine.com:60020
> 2011-07-28 08:17:54,006 [Thread-13] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=endSession.mde253811.preprod.ubithere.com,,00000000000000 for max=10 rows
> 2011-07-28 08:17:54,011 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for endSession.mde253811.preprod.ubithere.com,,1311086199483.706685579 is aws03.machine.com:60020
> 2011-07-28 08:17:54,017 [Thread-13] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=aws09.machine.com:2222 sessionTimeout=60000 watcher=hconnection
> 2011-07-28 08:17:54,017 [Thread-13-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server aws09.machine.com/10.83.1.244:2222
> 2011-07-28 08:17:54,018 [Thread-13-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to aws09.machine.com/10.83.1.244:2222, initiating session
> 2011-07-28 08:17:54,025 [Thread-13-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server aws09.machine.com/10.83.1.244:2222, sessionid = 0x131617dada6054d, negotiated timeout = 60000
> 2011-07-28 08:17:54,029 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@41f6321; hsa=aws03.machine.com:60020
> 2011-07-28 08:17:54,032 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for .META.,,1.1028785192 is aws03.machine.com:60020
> 2011-07-28 08:17:54,033 [Thread-13] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=endSession.mde253811.preprod.ubithere.com,,00000000000000 for max=10 rows
> 2011-07-28 08:17:54,037 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for endSession.mde253811.preprod.ubithere.com,,1311086199483.706685579 is aws03.machine.com:60020
> 2011-07-28 08:17:54,039 [Thread-13] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=endSession.mde253811.preprod.ubithere.com,,00000000000000 for max=2147483647 rows
> 2011-07-28 08:17:54,067 [Thread-13] DEBUG org.apache.hadoop.hbase.mapreduce.TableInputFormatBase - getSplits: split -> 0 -> aws03.machine.com:,
> 2011-07-28 08:17:54,068 [Thread-13] INFO org.apache.pig.backend.hadoop.hbase.HBaseTableInputFormat - Got 1 splits.
> 2011-07-28 08:17:54,068 [Thread-13] INFO org.apache.pig.backend.hadoop.hbase.HBaseTableInputFormat - Returning 1 splits.
> 2011-07-28 08:17:54,109 [Thread-13] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=aws09.machine.com:2222 sessionTimeout=60000 watcher=hconnection
> 2011-07-28 08:17:54,110 [Thread-13-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server aws09.machine.com/10.83.1.244:2222
> 2011-07-28 08:17:54,111 [Thread-13-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to aws09.machine.com/10.83.1.244:2222, initiating session
> 2011-07-28 08:17:54,119 [Thread-13-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server aws09.machine.com/10.83.1.244:2222, sessionid = 0x131617dada6054e, negotiated timeout = 60000
> 2011-07-28 08:17:54,123 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@20c3e967; hsa=aws03.machine.com:60020
> 2011-07-28 08:17:54,140 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for .META.,,1.1028785192 is aws03.machine.com:60020
> 2011-07-28 08:17:54,142 [Thread-13] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=startSession.mde253811.preprod.ubithere.com,,00000000000000 for max=10 rows
> 2011-07-28 08:17:54,148 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for startSession.mde253811.preprod.ubithere.com,,1311086198252.1334391323 is aws03.machine.com:60020
> 2011-07-28 08:17:54,154 [Thread-13] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=aws09.machine.com:2222 sessionTimeout=60000 watcher=hconnection
> 2011-07-28 08:17:54,158 [Thread-13-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server aws09.machine.com/10.83.1.244:2222
> 2011-07-28 08:17:54,159 [Thread-13-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to aws09.machine.com/10.83.1.244:2222, initiating session
> 2011-07-28 08:17:54,161 [Thread-13-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server aws09.machine.com/10.83.1.244:2222, sessionid = 0x131617dada6054f, negotiated timeout = 60000
> 2011-07-28 08:17:54,164 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5ee771f3; hsa=aws03.machine.com:60020
> 2011-07-28 08:17:54,167 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for .META.,,1.1028785192 is aws03.machine.com:60020
> 2011-07-28 08:17:54,169 [Thread-13] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=startSession.mde253811.preprod.ubithere.com,,00000000000000 for max=10 rows
> 2011-07-28 08:17:54,172 [Thread-13] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for startSession.mde253811.preprod.ubithere.com,,1311086198252.1334391323 is aws03.machine.com:60020
> 2011-07-28 08:17:54,173 [Thread-13] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=startSession.mde253811.preprod.ubithere.com,,00000000000000 for max=2147483647 rows
> 2011-07-28 08:17:54,180 [Thread-13] DEBUG org.apache.hadoop.hbase.mapreduce.TableInputFormatBase - getSplits: split -> 0 -> aws03.machine.com:,
> 2011-07-28 08:17:54,180 [Thread-13] INFO org.apache.pig.backend.hadoop.hbase.HBaseTableInputFormat - Got 1 splits.
> 2011-07-28 08:17:54,180 [Thread-13] INFO org.apache.pig.backend.hadoop.hbase.HBaseTableInputFormat - Returning 1 splits.
> 2011-07-28 08:17:55,037 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201107251336_0314
> 2011-07-28 08:17:55,037 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://aws09.preprod.ubithere.com:50030/jobdetails.jsp?jobid=job_201107251336_0314
> 2011-07-28 08:19:06,924 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 8% complete
> 2011-07-28 08:19:15,971 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 16% complete
> 2011-07-28 08:19:18,985 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 19% complete
> 2011-07-28 08:19:25,035 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 27% complete
> 2011-07-28 08:20:14,810 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
> 2011-07-28 08:20:14,812 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2011-07-28 08:20:14,830 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=aws09.machine.com:2222 sessionTimeout=60000 watcher=hconnection
> 2011-07-28 08:20:14,831 [main-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server aws09.machine.com/10.83.1.244:2222
> 2011-07-28 08:20:14,832 [main-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to aws09.machine.com/10.83.1.244:2222, initiating session
> 2011-07-28 08:20:14,838 [main-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server aws09.machine.com/10.83.1.244:2222, sessionid = 0x131617dada60556, negotiated timeout = 60000
> 2011-07-28 08:20:14,842 [main] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@5d4fa79d; hsa=aws03.machine.com:60020
> 2011-07-28 08:20:14,847 [main] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for .META.,,1.1028785192 is aws03.machine.com:60020
> 2011-07-28 08:20:14,849 [main] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=info.mde253811.preprod.ubithere.com,,00000000000000 for max=10 rows
> 2011-07-28 08:20:14,852 [main] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for info.mde253811.preprod.ubithere.com,,1311086202955.1975990008 is aws03.machine.com:60020
> 2011-07-28 08:20:16,311 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
> 2011-07-28 08:20:16,324 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=198330658
> 2011-07-28 08:20:16,324 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
> 2011-07-28 08:20:16,341 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
> 2011-07-28 08:20:16,656 [Thread-32] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
> 2011-07-28 08:20:16,656 [Thread-32] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> 2011-07-28 08:20:16,693 [Thread-32] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=aws09.machine.com:2222 sessionTimeout=60000 watcher=hconnection
> 2011-07-28 08:20:16,694 [Thread-32-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server aws09.machine.com/10.83.1.244:2222
> 2011-07-28 08:20:16,695 [Thread-32-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to aws09.machine.com/10.83.1.244:2222, initiating session
> 2011-07-28 08:20:16,702 [Thread-32-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server aws09.machine.com/10.83.1.244:2222, sessionid = 0x131617dada60557, negotiated timeout = 60000
> 2011-07-28 08:20:16,705 [Thread-32] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@37285252; hsa=aws03.machine.com:60020
> 2011-07-28 08:20:16,709 [Thread-32] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for .META.,,1.1028785192 is aws03.machine.com:60020
> 2011-07-28 08:20:16,710 [Thread-32] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=info.mde253811.preprod.ubithere.com,,00000000000000 for max=10 rows
> 2011-07-28 08:20:16,714 [Thread-32] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for info.mde253811.preprod.ubithere.com,,1311086202955.1975990008 is aws03.machine.com:60020
> 2011-07-28 08:20:16,716 [Thread-32] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=aws09.machine.com:2222 sessionTimeout=60000 watcher=hconnection
> 2011-07-28 08:20:16,717 [Thread-32-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server aws09.machine.com/10.83.1.244:2222
> 2011-07-28 08:20:16,718 [Thread-32-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to aws09.machine.com/10.83.1.244:2222, initiating session
> 2011-07-28 08:20:16,720 [Thread-32-SendThread(aws09.machine.com:2222)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server aws09.machine.com/10.83.1.244:2222, sessionid = 0x131617dada60558, negotiated timeout = 60000
> 2011-07-28 08:20:16,723 [Thread-32] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@7418e252; hsa=aws03.machine.com:60020
> 2011-07-28 08:20:16,726 [Thread-32] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for .META.,,1.1028785192 is aws03.machine.com:60020
> 2011-07-28 08:20:16,727 [Thread-32] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=info.mde253811.preprod.ubithere.com,,00000000000000 for max=10 rows
> 2011-07-28 08:20:16,730 [Thread-32] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation - Cached location for info.mde253811.preprod.ubithere.com,,1311086202955.1975990008 is aws03.machine.com:60020
> 2011-07-28 08:20:16,732 [Thread-32] DEBUG org.apache.hadoop.hbase.client.MetaScanner - Scanning .META. starting at row=info.mde253811.preprod.ubithere.com,,00000000000000 for max=2147483647 rows
> 2011-07-28 08:20:16,772 [Thread-32] DEBUG org.apache.hadoop.hbase.mapreduce.TableInputFormatBase - getSplits: split -> 0 -> aws03.machine.com:,
> 2011-07-28 08:20:16,772 [Thread-32] INFO org.apache.pig.backend.hadoop.hbase.HBaseTableInputFormat - Got 1 splits.
> 2011-07-28 08:20:16,772 [Thread-32] INFO org.apache.pig.backend.hadoop.hbase.HBaseTableInputFormat - Returning 1 splits.
> 2011-07-28 08:20:17,500 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201107251336_0315
> 2011-07-28 08:20:17,500 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://aws09.preprod.ubithere.com:50030/jobdetails.jsp?jobid=job_201107251336_0315
> 2011-07-28 08:20:28,075 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 37% complete
> 2011-07-28 08:20:34,106 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 41% complete
> 2011-07-28 08:20:37,124 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
> 2011-07-28 08:20:46,168 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 51% complete
> 2011-07-28 08:20:49,183 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 61% complete
> 2011-07-28 08:20:52,198 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 62% complete
> 2011-07-28 08:20:55,214 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 64% complete
> 2011-07-28 08:21:01,244 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 66% complete
> 2011-07-28 08:21:07,311 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
> 2011-07-28 08:21:07,312 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2011-07-28 08:21:08,770 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
> 2011-07-28 08:21:08,778 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
> 2011-07-28 08:21:08,910 [Thread-47] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
> 2011-07-28 08:21:08,910 [Thread-47] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> 2011-07-28 08:21:08,911 [Thread-47] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
> 2011-07-28 08:21:09,280 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_201107251336_0316
> 2011-07-28 08:21:09,280 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - More information at: http://aws09.preprod.ubithere.com:50030/jobdetails.jsp?jobid=job_201107251336_0316
> 2011-07-28 08:21:16,321 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 83% complete
> 2011-07-28 08:21:34,439 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
> 2011-07-28 08:21:34,441 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 0.20-append 0.8.1-SNAPSHOT root 2011-07-28 08:17:51 2011-07-28 08:21:34 HASH_JOIN,FILTER,LIMIT
> Success!
> Job Stats (time in seconds):
> JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime MaxReduceTime MinReduceTime AvgReduceTime Alias Feature Outputs
> job_201107251336_0314 2 1 75 66 70 63 63 63 end_sessions,sessions,start_sessions HASH_JOIN
> job_201107251336_0315 4 1 15 6 12 24 24 24 infos,sessions HASH_JOIN
> job_201107251336_0316 1 1 3 3 3 12 12 12 hdfs://aws09.preprod.ubithere.com:9000/tmp/temp-1404953096/tmp819396740,
> Input(s):
> Successfully read 2069446 records from: "endSession.mde253811.preprod.ubithere.com"
> Successfully read 2072419 records from: "startSession.mde253811.preprod.ubithere.com"
> Successfully read 19441 records from: "info.mde253811.preprod.ubithere.com"
> Output(s):
> Successfully stored 0 records in: "hdfs://aws09.preprod.ubithere.com:9000/tmp/temp-1404953096/tmp819396740"
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 1
> Total records proactively spilled: 1944943
> Job DAG:
> job_201107251336_0314 -> job_201107251336_0315,
> job_201107251336_0315 -> job_201107251336_0316,
> job_201107251336_0316
> 2011-07-28 08:21:34,472 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
> 2011-07-28 08:21:34,500 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
> 2011-07-28 08:21:34,501 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
> grunt>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira