You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Russell Jurney <ru...@gmail.com> on 2014/07/19 23:23:16 UTC
Unable to get pig local mode working
I downloaded pig 0.13, and I can't get local mode to work :(
14/07/19 14:22:05 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
14/07/19 14:22:05 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
2014-07-19 14:22:05,737 [main] INFO org.apache.pig.Main - Apache Pig
version 0.14.0-SNAPSHOT (rUnversioned directory) compiled Jul 19 2014,
14:07:36
2014-07-19 14:22:05,737 [main] INFO org.apache.pig.Main - Logging error
messages to: /private/tmp/pig_1405804925681.log
2014-07-19 14:22:05,952 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2014-07-19 14:22:05,952 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-07-19 14:22:05,953 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: file:///
2014-07-19 14:22:05,956 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.used.genericoptionsparser is deprecated. Instead, use
mapreduce.client.genericoptionsparser.used
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/Users/rjurney/Software/hadoop-2.0.0-cdh4.4.0/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Users/rjurney/Software/hbase-0.94.16-security/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
2014-07-19 14:22:06.063 java[6796:1903] Unable to load realm info from
SCDynamicStore
2014-07-19 14:22:06,365 [main] WARN
org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2014-07-19 14:22:06,581 [main] WARN org.apache.hadoop.conf.Configuration -
io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-07-19 14:22:06,583 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2014-07-19 14:22:06,583 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
grunt> emails = LOAD
'/Users/rjurney/Software/marketing/data/email_parts.txt' AS
(full_address:chararray, domain:chararray, name_part:chararray,
sld:chararray);
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.df.interval is deprecated. Instead, use fs.df.interval
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.task.tracker.http.address is deprecated. Instead, use
mapreduce.tasktracker.http.address
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.userlog.retain.hours is deprecated. Instead, use
mapreduce.job.userlog.retain.hours
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.local.dir.minspacestart is deprecated. Instead, use
mapreduce.tasktracker.local.dir.minspacestart
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.shuffle.read.timeout is deprecated. Instead, use
mapreduce.reduce.shuffle.read.timeout
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
io.sort.spill.percent is deprecated. Instead, use
mapreduce.map.sort.spill.percent
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.reduce.parallel.copies is deprecated. Instead, use
mapreduce.reduce.shuffle.parallelcopies
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.submit.replication is deprecated. Instead, use
mapreduce.client.submit.file.replication
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.local.dir.minspacekill is deprecated. Instead, use
mapreduce.tasktracker.local.dir.minspacekill
2014-07-19 14:22:12,158 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.task.profile is deprecated. Instead, use mapreduce.task.profile
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.heartbeats.in.second is deprecated. Instead, use
mapreduce.jobtracker.heartbeats.in.second
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.output.compress is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.healthChecker.interval is deprecated. Instead, use
mapreduce.tasktracker.healthchecker.interval
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.task.timeout is deprecated. Instead, use mapreduce.task.timeout
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.temp.dir is deprecated. Instead, use mapreduce.cluster.temp.dir
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
jobclient.completion.poll.interval is deprecated. Instead, use
mapreduce.client.completion.pollinterval
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker.persist.jobstatus.active is deprecated. Instead, use
mapreduce.jobtracker.persist.jobstatus.active
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.output.compression.codec is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress.codec
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.shuffle.merge.percent is deprecated. Instead, use
mapreduce.reduce.shuffle.merge.percent
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.map.max.attempts is deprecated. Instead, use
mapreduce.map.maxattempts
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.reduce.input.buffer.percent is deprecated. Instead, use
mapreduce.reduce.input.buffer.percent
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.task.cache.levels is deprecated. Instead, use
mapreduce.jobtracker.taskcache.levels
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.jobtracker.instrumentation is deprecated. Instead, use
mapreduce.jobtracker.instrumentation
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.userlog.limit.kb is deprecated. Instead, use
mapreduce.task.userlog.limit.kb
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.speculative.execution.slowNodeThreshold is deprecated. Instead, use
mapreduce.job.speculative.slownodethreshold
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.skip.map.max.skip.records is deprecated. Instead, use
mapreduce.map.skip.maxrecords
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker.jobhistory.lru.cache.size is deprecated. Instead, use
mapreduce.jobtracker.jobhistory.lru.cache.size
2014-07-19 14:22:12,159 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker.persist.jobstatus.hours is deprecated. Instead, use
mapreduce.jobtracker.persist.jobstatus.hours
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker.handler.count is deprecated. Instead, use
mapreduce.jobtracker.handler.count
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.task.profile.maps is deprecated. Instead, use
mapreduce.task.profile.maps
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.map.tasks.speculative.execution is deprecated. Instead, use
mapreduce.map.speculative
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.min.split.size is deprecated. Instead, use
mapreduce.input.fileinputformat.split.minsize
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.tasktracker.dns.nameserver is deprecated. Instead, use
mapreduce.tasktracker.dns.nameserver
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.tasktracker.taskmemorymanager.monitoring-interval is deprecated.
Instead, use mapreduce.tasktracker.taskmemorymanager.monitoringinterval
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.tasktracker.expiry.interval is deprecated. Instead, use
mapreduce.jobtracker.expire.trackers.interval
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.max.tracker.failures is deprecated. Instead, use
mapreduce.job.maxtaskfailures.per.tracker
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker.persist.jobstatus.dir is deprecated. Instead, use
mapreduce.jobtracker.persist.jobstatus.dir
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
job.end.retry.attempts is deprecated. Instead, use
mapreduce.job.end-notification.retry.attempts
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
mapreduce.reduce.speculative
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapreduce.job.counters.limit is deprecated. Instead, use
mapreduce.job.counters.max
2014-07-19 14:22:12,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.task.tracker.task-controller is deprecated. Instead, use
mapreduce.tasktracker.taskcontroller
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.jobtracker.maxtasks.per.job is deprecated. Instead, use
mapreduce.jobtracker.maxtasks.perjob
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.reduce.child.log.level is deprecated. Instead, use
mapreduce.reduce.log.level
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.reduce.max.attempts is deprecated. Instead, use
mapreduce.reduce.maxattempts
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.map.output.compression.codec is deprecated. Instead, use
mapreduce.map.output.compress.codec
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.shuffle.input.buffer.percent is deprecated. Instead, use
mapreduce.reduce.shuffle.input.buffer.percent
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.task.tracker.report.address is deprecated. Instead, use
mapreduce.tasktracker.report.address
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
keep.failed.task.files is deprecated. Instead, use
mapreduce.task.files.preserve.failedtasks
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
tasktracker.http.threads is deprecated. Instead, use
mapreduce.tasktracker.http.threads
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.speculative.execution.slowTaskThreshold is deprecated. Instead, use
mapreduce.job.speculative.slowtaskthreshold
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.acls.enabled is deprecated. Instead, use
mapreduce.cluster.acls.enabled
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.max.tracker.blacklists is deprecated. Instead, use
mapreduce.jobtracker.tasktracker.maxblacklists
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.tasktracker.indexcache.mb is deprecated. Instead, use
mapreduce.tasktracker.indexcache.mb
2014-07-19 14:22:12,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.skip.attempts.to.start.skipping is deprecated. Instead, use
mapreduce.task.skip.start.attempts
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.tasktracker.reduce.tasks.maximum is deprecated. Instead, use
mapreduce.tasktracker.reduce.tasks.maximum
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
jobclient.output.filter is deprecated. Instead, use
mapreduce.client.output.filter
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.jobtracker.restart.recover is deprecated. Instead, use
mapreduce.jobtracker.restart.recover
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.speculative.execution.speculativeCap is deprecated. Instead, use
mapreduce.job.speculative.speculativecap
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
jobclient.progress.monitor.poll.interval is deprecated. Instead, use
mapreduce.client.progressmonitor.pollinterval
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.map.child.log.level is deprecated. Instead, use
mapreduce.map.log.level
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.output.compression.type is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress.type
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker.retiredjobs.cache.size is deprecated. Instead, use
mapreduce.jobtracker.retiredjobs.cache.size
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.tasktracker.dns.interface is deprecated. Instead, use
mapreduce.tasktracker.dns.interface
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.task.profile.reduces is deprecated. Instead, use
mapreduce.task.profile.reduces
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
job.end.retry.interval is deprecated. Instead, use
mapreduce.job.end-notification.retry.interval
2014-07-19 14:22:12,162 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.jobtracker.job.history.block.size is deprecated. Instead, use
mapreduce.jobtracker.jobhistory.block.size
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.child.tmp is deprecated. Instead, use mapreduce.task.tmp.dir
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.tasktracker.map.tasks.maximum is deprecated. Instead, use
mapreduce.tasktracker.map.tasks.maximum
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use
mapreduce.job.committer.setup.cleanup.needed
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.queue.name is deprecated. Instead, use mapreduce.job.queuename
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.jobtracker.taskScheduler is deprecated. Instead, use
mapreduce.jobtracker.taskscheduler
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.skip.reduce.max.skip.groups is deprecated. Instead, use
mapreduce.reduce.skip.maxgroups
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker.http.address is deprecated. Instead, use
mapreduce.jobtracker.http.address
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.healthChecker.script.timeout is deprecated. Instead, use
mapreduce.tasktracker.healthchecker.script.timeout
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.tasktracker.instrumentation is deprecated. Instead, use
mapreduce.tasktracker.instrumentation
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.system.dir is deprecated. Instead, use
mapreduce.jobtracker.system.dir
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.reuse.jvm.num.tasks is deprecated. Instead, use
mapreduce.job.jvm.numtasks
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.inmem.merge.threshold is deprecated. Instead, use
mapreduce.reduce.merge.inmem.threshold
2014-07-19 14:22:12,163 [main] WARN org.apache.hadoop.conf.Configuration -
topology.script.number.args is deprecated. Instead, use
net.topology.script.number.args
2014-07-19 14:22:12,164 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.reduce.slowstart.completed.maps is deprecated. Instead, use
mapreduce.job.reduce.slowstart.completedmaps
2014-07-19 14:22:12,164 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.umaskmode is deprecated. Instead, use fs.permissions.umask-mode
2014-07-19 14:22:12,164 [main] WARN org.apache.hadoop.conf.Configuration -
topology.node.switch.mapping.impl is deprecated. Instead, use
net.topology.node.switch.mapping.impl
2014-07-19 14:22:12,164 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.tasktracker.tasks.sleeptime-before-sigkill is deprecated. Instead,
use mapreduce.tasktracker.tasks.sleeptimebeforesigkill
2014-07-19 14:22:12,164 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.compress.map.output is deprecated. Instead, use
mapreduce.map.output.compress
2014-07-19 14:22:12,164 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.merge.recordsBeforeProgress is deprecated. Instead, use
mapreduce.task.merge.progress.records
2014-07-19 14:22:12,164 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.shuffle.connect.timeout is deprecated. Instead, use
mapreduce.reduce.shuffle.connect.timeout
2014-07-19 14:22:12,164 [main] WARN org.apache.hadoop.conf.Configuration -
io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
grunt>
grunt> name_counts = FOREACH (GROUP emails BY name_part) GENERATE group AS
name_part, COUNT_STAR(emails) AS total;
grunt> sorted_counts = ORDER name_counts BY total DESC;
grunt> top_1000 = LIMIT sorted_counts 1000;
grunt> dump top_1000
2014-07-19 14:22:12,639 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: GROUP_BY,ORDER_BY,LIMIT
2014-07-19 14:22:12,725 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter,
LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach,
PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter,
SplitFilter, StreamTypeCastInserter],
RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2014-07-19 14:22:12,911 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2014-07-19 14:22:12,933 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil -
Choosing to move algebraic foreach to combiner
2014-07-19 14:22:12,956 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.SecondaryKeyOptimizerMR
- Using Secondary Key Optimization for MapReduce node scope-30
2014-07-19 14:22:12,967 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 4
2014-07-19 14:22:12,967 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 4
2014-07-19 14:22:13,004 [main] WARN org.apache.hadoop.conf.Configuration -
io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-07-19 14:22:13,004 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2014-07-19 14:22:13,062 [main] WARN org.apache.hadoop.conf.Configuration -
session.id is deprecated. Instead, use dfs.metrics.session-id
2014-07-19 14:22:13,062 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId=
2014-07-19 14:22:13,138 [main] WARN
org.apache.pig.backend.hadoop20.PigJobControl - falling back to default
JobControl (not using hadoop 0.20 ?)
java.lang.NoSuchFieldException: runnerState
at java.lang.Class.getDeclaredField(Class.java:1938)
at
org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51)
at
org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:106)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:313)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:199)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1378)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363)
at org.apache.pig.PigServer.storeEx(PigServer.java:1022)
at org.apache.pig.PigServer.store(PigServer.java:985)
at org.apache.pig.PigServer.openIterator(PigServer.java:898)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:753)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:229)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:204)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:543)
at org.apache.pig.Main.main(Main.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
2014-07-19 14:22:13,141 [main] INFO
org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings
are added to the job
2014-07-19 14:22:13,154 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-07-19 14:22:13,154 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.output.compress is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress
2014-07-19 14:22:13,156 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Reduce phase detected, estimating # of required reducers.
2014-07-19 14:22:13,156 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Using reducer estimator:
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2014-07-19 14:22:13,161 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
- BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=30508407
2014-07-19 14:22:13,161 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting Parallelism to 1
2014-07-19 14:22:13,161 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2014-07-19 14:22:13,196 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2014-07-19 14:22:13,202 [main] INFO
org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false,
will not generate code.
2014-07-19 14:22:13,202 [main] INFO
org.apache.pig.data.SchemaTupleFrontend - Starting process to move
generated code to distributed cacche
2014-07-19 14:22:13,202 [main] INFO
org.apache.pig.data.SchemaTupleFrontend - Distributed cache not supported
or needed in local mode. Setting key [pig.schematuple.local.dir] with code
temp directory:
/var/folders/0b/74l_65015_5fcbmbdz1w2xl40000gn/T/1405804933202-0
2014-07-19 14:22:13,324 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2998: Unhandled internal error.
org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
2014-07-19 14:22:13,324 [main] ERROR org.apache.pig.tools.grunt.Grunt -
java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:324)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:199)
at
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1378)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1363)
at org.apache.pig.PigServer.storeEx(PigServer.java:1022)
at org.apache.pig.PigServer.store(PigServer.java:985)
at org.apache.pig.PigServer.openIterator(PigServer.java:898)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:753)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:229)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:204)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
at org.apache.pig.Main.run(Main.java:543)
at org.apache.pig.Main.main(Main.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Details also at logfile: /private/tmp/pig_1405804925681.log
--
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com
ᐧ
Re: Unable to get pig local mode working
Posted by Russell Jurney <ru...@gmail.com>.
The problem was resolve by:
HADOOP_CLIENT_OPTS="-Xmx1024m"
ᐧ
On Sat, Jul 19, 2014 at 4:24 PM, Cheolsoo Park <pi...@gmail.com> wrote:
> My guess is that you have Hadoop 2 in your classpath and are using Hadoop
> 1-compile pig-withouthadoop jar.
>
> Btw, Pig 0.13 started shipping both h1 and h2 jars and made Pig invoking
> script decide which version to use based on the existence of
> hadoop-core.jar in classpath. That might be causing the error to you. See
> details here-
> https://issues.apache.org/jira/browse/PIG-3892
>
--
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com
Re: Unable to get pig local mode working
Posted by Cheolsoo Park <pi...@gmail.com>.
My guess is that you have Hadoop 2 in your classpath and are using Hadoop
1-compile pig-withouthadoop jar.
Btw, Pig 0.13 started shipping both h1 and h2 jars and made Pig invoking
script decide which version to use based on the existence of
hadoop-core.jar in classpath. That might be causing the error to you. See
details here-
https://issues.apache.org/jira/browse/PIG-3892