You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ramgopal N (Created) (JIRA)" <ji...@apache.org> on 2011/10/17 12:02:11 UTC
[jira] [Created] (MAPREDUCE-3193) NextGen Mapreduce framework is
not able to read the job input recursively.Input is read only for one
folder level deep
NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
----------------------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-3193
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramgopal N
java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Status: Patch Available (was: Open)
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Attachment: MAPREDUCE-3193-3.patch
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.2, 0.23.2, 2.0.0-alpha, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-3.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Status: Open (was: Patch Available)
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 2.0.0-alpha, 1.0.2, 0.23.2, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) NextGen Mapreduce framework is
not able to read the job input recursively.Input is read only for one
folder level deep
Posted by "Devaraj K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130382#comment-13130382 ]
Devaraj K commented on MAPREDUCE-3193:
--------------------------------------
bq. This isn't something we should change lightly, it's probably going to break user apps.
If the input path contains one nested dir, it is considering as file and trying to execute the task and it fails with the below error. Failing the job itself when the inputpath contains nested dir might not be correct.
{code:xml}
Caused by: java.io.FileNotFoundException: File does not exist: /r1/r2
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:736)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:699)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:671)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:315)
at org.apache.hadoop.hdfs.protocolR23Compatible.ClientNamenodeProtocolServerSideTranslatorR23.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorR23.java:130)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:632)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1517)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1513)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1511)
at org.apache.hadoop.ipc.Client.call(Client.java:1085)
at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244)
at $Proxy8.getBlockLocations(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:130)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:81)
at $Proxy8.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.protocolR23Compatible.ClientNamenodeProtocolTranslatorR23.getBlockLocations(ClientNamenodeProtocolTranslatorR23.java:150)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:566)
... 14 more
{code}
> NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157928#comment-13157928 ]
Devaraj K commented on MAPREDUCE-3193:
--------------------------------------
bq.-1 javac. The applied patch generated 1818 javac compiler warnings (more than the trunk's current 1817 warnings).
It is due to the deprecated MiniDFSCluster used in new test added for TestFileInputFormat.
bq. -1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings.
These warnings are not introduced by this patch.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257356#comment-13257356 ]
Devaraj K commented on MAPREDUCE-3193:
--------------------------------------
It has been in 'Patch Available' for some time and good to have it in.
Can any one have a look into the patch and review it? Thnx.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.2, 2.0.0, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) NextGen Mapreduce framework is
not able to read the job input recursively.Input is read only for one
folder level deep
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Attachment: MAPREDUCE-3193.patch
> NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285410#comment-13285410 ]
Harsh J commented on MAPREDUCE-3193:
------------------------------------
Thanks for the patches. This should get in soon cause its a wide divide between old and new API feature sets.
I have a few questions though:
bq. MAPREDUCE-1501 added this behaviour to the old API. Can you change your patch to share code and tests so that both the old and new API behave in the same way? Also, the old configuration parameter should be deprecated, but still supported in the new API.
Given that both APIs are now supported, do we really need the deprecation? Will the new name apply to both? Are other properties handled in the same way today?
For example I see in old API the following reuse:
{code}
public static final String NUM_INPUT_FILES =
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.NUM_INPUT_FILES;
{code}
While this patch does not change similar things in mapred.lib even after deprecation marker. Can this be done here too?
{quote}
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-hdfs</artifactId>
{quote}
Can the test not be done with just LFS? We can avoid a dependency if it can be done. Similarly a LJRunner test would be great too, if alright - instead of an MR cluster.
bq. mapreduce.input.fileinputformat.readinputfilesrecursively
The last part can still be bettered I think. (Nit: Its not reading recursively, just listing that way.) Perhaps "mapreduce.input.fileinputformat.input.dir.recursive" is simpler to have?
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.2, 0.23.2, 2.0.0-alpha, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Affects Version/s: (was: 0.23.0)
0.24.0
Summary: FileInputFormat doesn't read files recursively in the input path dir (was: NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep)
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) NextGen Mapreduce framework is
not able to read the job input recursively.Input is read only for one
folder level deep
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129001#comment-13129001 ]
Mahadev konar commented on MAPREDUCE-3193:
------------------------------------------
Ramgopal,
What input format are you using?
> NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramgopal N
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285609#comment-13285609 ]
Harsh J commented on MAPREDUCE-3193:
------------------------------------
bq. Thanks Harsh for looking into the patch. Thanks a lot for your time.
I've been busy lately, but I plan to go over your other patches soon. Sorry for the delays on your requests!
bq. I don't see any problem of having the test cases this way other than adding the dependency. I feel would be good to have this test case with MiniDFSCluster.
The problem is test-run time. With LFS/LJR its pretty fast, and if test isn't HDFS specific nor has anything to do with MR framework in particular, we can avoid using mini clusters.
bq. I thought from the external perspective, this we don't use other than for reading purpose. Anyway we can change this name to "mapreduce.input.fileinputformat.input.dir.recursive". I will update in the next patch.
Config elements are also utilized by streaming users since there's no other way for them other than to use -D to specify this. Readability helps in general, wherever it is :)
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.2, 0.23.2, 2.0.0-alpha, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132736#comment-13132736 ]
Mahadev konar commented on MAPREDUCE-3193:
------------------------------------------
Devaraj,
Can you please run ant test with the patch? There are quite a few tests for FileInputForamt in ant which are not run via the hudson bot.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131685#comment-13131685 ]
Devaraj K commented on MAPREDUCE-3193:
--------------------------------------
bq. -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.
These findbugs are not related to the patch.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285540#comment-13285540 ]
Devaraj K commented on MAPREDUCE-3193:
--------------------------------------
Thanks Harsh for looking into the patch. Thanks a lot for your time.
{quote}
Given that both APIs are now supported, do we really need the deprecation? Will the new name apply to both? Are other properties handled in the same way today?
For example I see in old API the following reuse:
{code:xml}
public static final String NUM_INPUT_FILES =
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.NUM_INPUT_FILES;
{code}
While this patch does not change similar things in mapred.lib even after deprecation marker. Can this be done here too?
{quote}
The deprecation is still needed. I should use the new name reuse in the old api as it is done for other properties. I will update in the next patch.
bq. Can the test not be done with just LFS? We can avoid a dependency if it can be done. Similarly a LJRunner test would be great too, if alright - instead of an MR cluster.
I don't see any problem of having the test cases this way other than adding the dependency. I feel would be good to have this test case with MiniDFSCluster.
bq. The last part can still be bettered I think. (Nit: Its not reading recursively, just listing that way.) Perhaps "mapreduce.input.fileinputformat.input.dir.recursive" is simpler to have?
I thought from the external perspective, this we don't use other than for reading purpose. Anyway we can change this name to "mapreduce.input.fileinputformat.input.dir.recursive". I will update in the next patch.
Please let me if any other nits/suggestions, I will update the patch.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.2, 0.23.2, 2.0.0-alpha, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Attachment: MAPREDUCE-3193-2.patch
Resubmitting the patch to trigger hudson with patch on trunk branch.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134272#comment-13134272 ]
Tom White commented on MAPREDUCE-3193:
--------------------------------------
MAPREDUCE-1501 added this behaviour to the old API. Can you change your patch to share code and tests so that both the old and new API behave in the same way? Also, the old configuration parameter should be deprecated, but still supported in the new API.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Component/s: mrv1
Affects Version/s: (was: 0.24.0)
3.0.0
2.0.0
1.0.2
Fix Version/s: (was: 1.1.0)
(was: 0.24.0)
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.2, 2.0.0, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Fix Version/s: 1.1.0
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0, 1.1.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427990#comment-13427990 ]
Devaraj K commented on MAPREDUCE-3193:
--------------------------------------
Hi Harsh, can you have a look into the updated patch when you find some time?
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.2, 0.23.2, 2.0.0-alpha, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-3.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) NextGen Mapreduce framework is
not able to read the job input recursively.Input is read only for one
folder level deep
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130372#comment-13130372 ]
Arun C Murthy commented on MAPREDUCE-3193:
------------------------------------------
This isn't something we should change lightly, it's probably going to break user apps.
At least, we need a config to turn this on, and it should be off by default.
> NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) NextGen Mapreduce framework is
not able to read the job input recursively.Input is read only for one
folder level deep
Posted by "Ramgopal N (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129481#comment-13129481 ]
Ramgopal N commented on MAPREDUCE-3193:
---------------------------------------
I have executed Wordcount job from examples.jar.It uses FileInputFormat
> NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramgopal N
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286517#comment-13286517 ]
Hadoop QA commented on MAPREDUCE-3193:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530373/MAPREDUCE-3193-3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2426//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2426//console
This message is automatically generated.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.2, 0.23.2, 2.0.0-alpha, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-3.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Status: Open (was: Patch Available)
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Status: Patch Available (was: Open)
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) NextGen Mapreduce framework is
not able to read the job input recursively.Input is read only for one
folder level deep
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Attachment: MAPREDUCE-3193-1.patch
> NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131683#comment-13131683 ]
Hadoop QA commented on MAPREDUCE-3193:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499866/MAPREDUCE-3193-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1082//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1082//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1082//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1082//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1082//console
This message is automatically generated.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157135#comment-13157135 ]
Hadoop QA commented on MAPREDUCE-3193:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12505105/MAPREDUCE-3193.security.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1343//console
This message is automatically generated.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) NextGen Mapreduce framework is
not able to read the job input recursively.Input is read only for one
folder level deep
Posted by "Devaraj K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129630#comment-13129630 ]
Devaraj K commented on MAPREDUCE-3193:
--------------------------------------
Here is the problem. In FileInputFormat.listStatus, It considers the files/directories in one nested level and takes every thing as file. Finally it creates splits with directories and fails the task.
{code:title=FileInputFormat.java|borderStyle=solid}
for (int i=0; i < dirs.length; ++i) {
Path p = dirs[i];
FileSystem fs = p.getFileSystem(job.getConfiguration());
FileStatus[] matches = fs.globStatus(p, inputFilter);
if (matches == null) {
errors.add(new IOException("Input path does not exist: " + p));
} else if (matches.length == 0) {
errors.add(new IOException("Input Pattern " + p + " matches 0 files"));
} else {
for (FileStatus globStat: matches) {
if (globStat.isDirectory()) {
for(FileStatus stat: fs.listStatus(globStat.getPath(),
inputFilter)) {
result.add(stat);
}
} else {
result.add(globStat);
}
}
}
}
{code}
> NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramgopal N
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Target Version/s: 2.0.1-alpha, 3.0.0
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.2, 0.23.2, 2.0.0-alpha, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-3.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132766#comment-13132766 ]
Devaraj K commented on MAPREDUCE-3193:
--------------------------------------
Hi Mahdev,
I ran ant test, there are no failures because of this patch. Some tests fail in my env always which are not related to this.
Anyway it uses the old behavior by default, if we want to read recursively need to set that property value as true.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157530#comment-13157530 ]
Hadoop QA commented on MAPREDUCE-3193:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12505206/MAPREDUCE-3193-2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1818 javac compiler warnings (more than the trunk's current 1817 warnings).
-1 findbugs. The patch appears to introduce 12 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1344//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1344//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-examples.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1344//console
This message is automatically generated.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3193:
-------------------------------------
Status: Open (was: Patch Available)
cancelling patch to address Tom's comments.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Status: Patch Available (was: Open)
Updated the patch for trunk.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 2.0.0-alpha, 1.0.2, 0.23.2, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-3.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153314#comment-13153314 ]
Devaraj K commented on MAPREDUCE-3193:
--------------------------------------
Yes Prashant. I will provide a patch for 0.20.205 also. Thanks.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Attachment: MAPREDUCE-3193.security.patch
MAPREDUCE-3193-2.patch
Attached the patch for trunk and security branch ,addressing the review comments.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Fix Version/s: 0.24.0
Status: Patch Available (was: Open)
I updated the patch by making this change as configurable, by default uses the old behaviour.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3193) NextGen Mapreduce framework is
not able to read the job input recursively.Input is read only for one
folder level deep
Posted by "Devaraj K (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K reassigned MAPREDUCE-3193:
------------------------------------
Assignee: Devaraj K
> NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) NextGen Mapreduce framework is
not able to read the job input recursively.Input is read only for one
folder level deep
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130384#comment-13130384 ]
Arun C Murthy commented on MAPREDUCE-3193:
------------------------------------------
bq. If the input path contains one nested dir, it is considering as file and trying to execute the task and it fails with the below error. Failing the job itself when the inputpath contains nested dir might not be correct.
Yes, I realize that.
FileInputFormat has this behaviour for a long while and changing it now (we definitely shouldn't do this for hadoop-0.23.0) will probably affect a lot of apps. Hence, at the very least, we need to have this off by default.
> NextGen Mapreduce framework is not able to read the job input recursively.Input is read only for one folder level deep
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Devaraj K (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated MAPREDUCE-3193:
---------------------------------
Affects Version/s: 0.23.2
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 0.23.2, 1.0.2, 2.0.0, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Shilo Ayalon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257331#comment-13257331 ]
Shilo Ayalon commented on MAPREDUCE-3193:
-----------------------------------------
I'm having the same problem with hadoop-1.0.2. Given the following directory structure (in hdfs):
{code}
one/
├── three/
│ └── four/
│ ├── baz.txt
│ ├── bleh.txt
│ └── foo.txt
└── two/
├── bar.txt
└── gaa.txt
{code}
As no recursive path support is available, I'm walking the root folder and adding all subdirs to the job. However, adding file-less folders like {{one}} and {{one/three}} as input paths to the job raises this exception:
{code}
java.io.FileNotFoundException: /user/hduser/data/one (Is a directory)
{code}
The actual number of files present is massive (20k/30k+), so passing all on the command line seems redundant. Will this patch be added to the source at some point?
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
> Affects Versions: 1.0.2, 2.0.0, 3.0.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3193) FileInputFormat doesn't read
files recursively in the input path dir
Posted by "Prashant Sharma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152723#comment-13152723 ]
Prashant Sharma commented on MAPREDUCE-3193:
--------------------------------------------
Folks is it WIP? can you also provide fix for 20.205.
> FileInputFormat doesn't read files recursively in the input path dir
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3193
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.24.0
> Reporter: Ramgopal N
> Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193.patch
>
>
> java.io.FileNotFoundException is thrown,if input file is more than one folder level deep and the job is getting failed.
> Example:Input file is /r1/r2/input.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira