You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mark <st...@gmail.com> on 2011/04/06 16:53:44 UTC
Setting input paths
How can I tell my job to include all the subdirectories and their
content of a certain path?
My directory structure is as follows: logs/{YEAR}/{MONTH}/{DAY} and I
tried setting my input path to 'logs/' using
FileInputFormat.addInputPath however I keep receiving the following error:
ava.io.FileNotFoundException: File does not exist: /user/root/logs/2011/01
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1586)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1577)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:428)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:234
Do my directories/directory contents need to be in any particular format?
Thanks
Re: Setting input paths
Posted by hadoopman <ha...@gmail.com>.
I have a process which is loading data into hive hourly. Loading data
hourly isn't a problem however when I load historical data say 24-48
hours I receive the below error msg. In googling I've come across some
suggestions that jvm memory needs to be increased. Are there any other
options or is that pretty much it?
I appreciate any help with this one. To get around the problem I've
loaded fewer historical logs which works great but isn't what I had in
mind :-)
Thanks a bunch!
---
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 finished. closing...
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 forwarded 0 rows
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 finished. closing...
2011-04-05 15:33:38,400 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 forwarded 0 rows
2011-04-05 15:33:38,401 WARN
org.apache.hadoop.hive.ql.exec.GroupByOperator: Begin Hash Table flush
at close: size = 0
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: 9 finished. closing...
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: 9 forwarded 0 rows
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: Final Path: FS
hdfs://namenoden1:9000
/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/000049_0
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: Writing to temp file: FS
hdfs://namenoden1:9000/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/_tmp.000049_0
2011-04-05 15:33:38,401 INFO
org.apache.hadoop.hive.ql.exec.FileSinkOperator: New Final Path: FS
hdfs://namenoden1:9000
/tmp/hive-etl/hive_2011-04-05_15-25-32_126_7118636463039801851/_tmp.-mr-10004/000049_0
2011-04-05 15:33:38,448 INFO org.apache.hadoop.io.compress.CodecPool:
Got brand-new compressor
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: 8 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 7 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.FilterOperator: 6 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.SelectOperator: 1 Close done
2011-04-05 15:33:38,495 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 Close done
2011-04-05 15:33:38,495 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
13 Close done
2011-04-05 15:33:38,495 INFO ExecMapper: ExecMapper: processed 188614
rows: used memory = 720968432
2011-04-05 15:33:38,501 FATAL org.apache.hadoop.mapred.Child: Error
running child : java.lang.OutOfMemoryError: Java
heap space
at org.apache.hadoop.io.Text.setCapacity(Text.java:240)
at org.apache.hadoop.io.Text.append(Text.java:216)
at
org.apache.hadoop.util.LineReader.readLine(LineReader.java:159)
at
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136)
at
org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40)
at
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:66)
at
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:32)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:67)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:202)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:186)
at
org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:383)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:317)
at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
at java.security.AccessController.doPrivileged(Native
Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
at org.apache.hadoop.mapred.Child.main(Child.java:211)
Re: Setting input paths
Posted by Harsh Chouraria <ha...@cloudera.com>.
Hello Mark and Robert,
On Wed, Apr 6, 2011 at 9:55 PM, Mark <st...@gmail.com> wrote:
>> On 4/6/11 9:53 AM, "Mark"<st...@gmail.com> wrote:
>>
>> How can I tell my job to include all the subdirectories and their
>> content of a certain path?
Also worth noting is that in future releases of Hadoop, it will be
possible to ask the FileInputFormat itself to recursively add file
entries (or add till specified depth levels). MAPREDUCE-1501 covered
this feature.
--
Harsh J
Support Engineer, Cloudera
http://www.cloudera.com/
Re: Setting input paths
Posted by Mark <st...@gmail.com>.
Ok so the behavior is a little different when using
FileInputFormat.addInputPath
as opposed to using pig. Ill try the glob.
Thanks
On 4/6/11 8:41 AM, Robert Evans wrote:
> I believe that opening a directory as a file will result in a file not found. You probably need to set it to a glob, that points to that actual files. Something like
>
> /user/root/logs/2011/*/*/* for all entries in 2011, or /user/root/logs/2011/01/*/* if you want to restrict it to just January. By default if you pass in a directory as input the input format will assume that the directory contains only files, no sub directories and that you really want to use each of those files an input.
>
> --Bobby Evans
>
> On 4/6/11 9:53 AM, "Mark"<st...@gmail.com> wrote:
>
> How can I tell my job to include all the subdirectories and their
> content of a certain path?
>
> My directory structure is as follows: logs/{YEAR}/{MONTH}/{DAY} and I
> tried setting my input path to 'logs/' using
> FileInputFormat.addInputPath however I keep receiving the following error:
>
> ava.io.FileNotFoundException: File does not exist: /user/root/logs/2011/01
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1586)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1577)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:428)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
> at org.apache.hadoop.mapred.Child.main(Child.java:234
>
>
> Do my directories/directory contents need to be in any particular format?
>
> Thanks
>
>
Re: Setting input paths
Posted by Robert Evans <ev...@yahoo-inc.com>.
I believe that opening a directory as a file will result in a file not found. You probably need to set it to a glob, that points to that actual files. Something like
/user/root/logs/2011/*/*/* for all entries in 2011, or /user/root/logs/2011/01/*/* if you want to restrict it to just January. By default if you pass in a directory as input the input format will assume that the directory contains only files, no sub directories and that you really want to use each of those files an input.
--Bobby Evans
On 4/6/11 9:53 AM, "Mark" <st...@gmail.com> wrote:
How can I tell my job to include all the subdirectories and their
content of a certain path?
My directory structure is as follows: logs/{YEAR}/{MONTH}/{DAY} and I
tried setting my input path to 'logs/' using
FileInputFormat.addInputPath however I keep receiving the following error:
ava.io.FileNotFoundException: File does not exist: /user/root/logs/2011/01
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1586)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1577)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:428)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:645)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:240)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.hadoop.mapred.Child.main(Child.java:234
Do my directories/directory contents need to be in any particular format?
Thanks