You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Raviteja Chirala (JIRA)" <ji...@apache.org> on 2013/09/30 06:10:26 UTC
[jira] [Commented] (PIG-3272) While running a pig job,showing delay
after Snappy native library get loaded
[ https://issues.apache.org/jira/browse/PIG-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781605#comment-13781605 ]
Raviteja Chirala commented on PIG-3272:
---------------------------------------
Hi [~sambit], I am also not able to reproduce this issue in my machine. I am using pig 0.11.1.
2013-09-29 22:00:29,549 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 200
2013-09-29 22:00:29,646 INFO org.apache.hadoop.mapred.MapTask: data buffer = 150994944/167772160
2013-09-29 22:00:29,646 INFO org.apache.hadoop.mapred.MapTask: record buffer = 2359296/2621440
2013-09-29 22:00:29,672 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
2013-09-29 22:00:29,675 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev cf4e7cbf8ed0f0622504d008101c2729dc0c9ff3]
2013-09-29 22:00:29,680 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is available
2013-09-29 22:00:29,680 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library loaded
2013-09-29 22:00:29,760 INFO org.apache.pig.data.SchemaTupleBackend: Key [pig.schematuple] was not set... will not generate code.
2013-09-29 22:00:30,004 INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map: Aliases being processed per job phase
2013-09-29 22:05:27,526 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2013-09-29 22:05:27,532 INFO org.apache.hadoop.mapred.Task: Task:attempt_201309170004_0160_m_000119_0 is done. And is in the process of commiting
2013-09-29 22:05:27,722 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201309170004_0160_m_000119_0' done.
> While running a pig job,showing delay after Snappy native library get loaded
> ----------------------------------------------------------------------------
>
> Key: PIG-3272
> URL: https://issues.apache.org/jira/browse/PIG-3272
> Project: Pig
> Issue Type: Bug
> Components: data
> Affects Versions: 0.10.1
> Reporter: Sambit Sahoo
>
> 2013-04-09 06:29:42,639 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
> 2013-04-09 06:29:43,947 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 1024
> 2013-04-09 06:29:44,434 INFO org.apache.hadoop.mapred.MapTask: data buffer = 816043776/1020054736
> 2013-04-09 06:29:44,435 INFO org.apache.hadoop.mapred.MapTask: record buffer = 2684354/3355443
> 2013-04-09 06:29:44,448 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is available
> 2013-04-09 06:29:44,448 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library loaded
> 2013-04-09 06:48:29,783 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
> 2013-04-09 06:48:29,783 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 423372603; bufvoid = 1020054736
> 2013-04-09 06:48:29,783 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 2684354; length = 3355443
> 2013-04-09 06:49:32,969 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor
> In the above log you will see, after Snappy native library loaded it took around 20mins to proceed.
> Can anyone tell me the issue and the resolution?
> is it due to some network issue?
--
This message was sent by Atlassian JIRA
(v6.1#6144)