You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Felix Gao (JIRA)" <ji...@apache.org> on 2011/03/03 00:40:36 UTC
[jira] Updated: (PIG-1879) AvroStorage with wildcard causes extreme
slow initialization
[ https://issues.apache.org/jira/browse/PIG-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Felix Gao updated PIG-1879:
---------------------------
Description:
I am using piggybank with avro_storage.patch from PIG-1748. When I load the data using Load '/user/test/logs/avro/\*/\*' USING org.apache.pig.piggybank.storage.avro.AvroStorage(); The sys log on the mapper shows
2011-03-02 12:52:35,556 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2011-03-02 12:52:37,333 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
However, when I use load '/user/test/logs/avro/{2011-03-01/*/*-23-00*,2011-03-01/*/*2011-03-01-00-00*,2011-03-01/*/*2011-03-01-01-00*,2011-03-01/*/*2011-03-01-02-00*,2011-03-01/*/*2011-03-01-03-00*,2011-03-01/*/*2011-03-01-04-00*,2011-02-28/*/*2011-02-28-05-00*,2011-02-28/*/*2011-02-28-06-00*,2011-02-28/*/*2011-02-28-07-00*,2011-02-28/*/*2011-02-28-08-00*,2011-02-28/*/*2011-02-28-09-00*,2011-02-28/*/*2011-02-28-1*-00*,2011-02-28/*/*2011-02-28-20-00*,2011-02-28/*/*2011-02-28-21-00*,2011-02-28/*/*2011-02-28-22-00*}' The sys log on the mapper shows
2011-03-02 12:03:33,091 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2011-03-02 12:06:05,254 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
Notice it took 2 minute and 30 seconds on the initialization stage. If I cut the number of file patterns in the glob to half. The mappers will be twice as fast.
was:
I am using piggybank with avro_storage.patch from PIG-1748. When I load the data using Load '/user/test/logs/avro/*/*' USING org.apache.pig.piggybank.storage.avro.AvroStorage(); The sys log on the mapper shows
2011-03-02 12:52:35,556 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2011-03-02 12:52:37,333 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
However, when I use load '/user/test/logs/avro/{2011-03-01/*/*-23-00*,2011-03-01/*/*2011-03-01-00-00*,2011-03-01/*/*2011-03-01-01-00*,2011-03-01/*/*2011-03-01-02-00*,2011-03-01/*/*2011-03-01-03-00*,2011-03-01/*/*2011-03-01-04-00*,2011-02-28/*/*2011-02-28-05-00*,2011-02-28/*/*2011-02-28-06-00*,2011-02-28/*/*2011-02-28-07-00*,2011-02-28/*/*2011-02-28-08-00*,2011-02-28/*/*2011-02-28-09-00*,2011-02-28/*/*2011-02-28-1*-00*,2011-02-28/*/*2011-02-28-20-00*,2011-02-28/*/*2011-02-28-21-00*,2011-02-28/*/*2011-02-28-22-00*}' The sys log on the mapper shows
2011-03-02 12:03:33,091 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
2011-03-02 12:06:05,254 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
Notice it took 2 minute and 30 seconds on the initialization stage. If I cut the number of file patterns in the glob to half. The mappers will be twice as fast.
> AvroStorage with wildcard causes extreme slow initialization
> ------------------------------------------------------------
>
> Key: PIG-1879
> URL: https://issues.apache.org/jira/browse/PIG-1879
> Project: Pig
> Issue Type: Bug
> Components: tools
> Affects Versions: 0.7.0
> Reporter: Felix Gao
>
> I am using piggybank with avro_storage.patch from PIG-1748. When I load the data using Load '/user/test/logs/avro/\*/\*' USING org.apache.pig.piggybank.storage.avro.AvroStorage(); The sys log on the mapper shows
> 2011-03-02 12:52:35,556 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
> 2011-03-02 12:52:37,333 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
> However, when I use load '/user/test/logs/avro/{2011-03-01/*/*-23-00*,2011-03-01/*/*2011-03-01-00-00*,2011-03-01/*/*2011-03-01-01-00*,2011-03-01/*/*2011-03-01-02-00*,2011-03-01/*/*2011-03-01-03-00*,2011-03-01/*/*2011-03-01-04-00*,2011-02-28/*/*2011-02-28-05-00*,2011-02-28/*/*2011-02-28-06-00*,2011-02-28/*/*2011-02-28-07-00*,2011-02-28/*/*2011-02-28-08-00*,2011-02-28/*/*2011-02-28-09-00*,2011-02-28/*/*2011-02-28-1*-00*,2011-02-28/*/*2011-02-28-20-00*,2011-02-28/*/*2011-02-28-21-00*,2011-02-28/*/*2011-02-28-22-00*}' The sys log on the mapper shows
> 2011-03-02 12:03:33,091 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
> 2011-03-02 12:06:05,254 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
> Notice it took 2 minute and 30 seconds on the initialization stage. If I cut the number of file patterns in the glob to half. The mappers will be twice as fast.
>
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira