You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Mateusz Berezecki (JIRA)" <ji...@apache.org> on 2008/08/25 21:47:44 UTC
[jira] Created: (HADOOP-4020) Hadoop Pipes do not load custom
InputFormats at appropriate time, rendering them useless in certain
scenarios
Hadoop Pipes do not load custom InputFormats at appropriate time, rendering them useless in certain scenarios
-------------------------------------------------------------------------------------------------------------
Key: HADOOP-4020
URL: https://issues.apache.org/jira/browse/HADOOP-4020
Project: Hadoop Core
Issue Type: Improvement
Components: pipes
Affects Versions: 0.17.2, 0.17.1, 0.17.0, 0.16.4
Reporter: Mateusz Berezecki
There's a patch provided below.
It was created on the 0.17.0 version,
but with a little tweaking can be applied
to the 0.18.0 as well. It fixes a timing bug,
whereas a custom inputformat was loaded
from the jar file, but the inputformat was being
processed much earlier in time, resulting
in a class not found exception.
--- src/java/org/apache/hadoop/mapred/pipes/Submitter.java.orig 2008-05-15 09:20:16.000000000 +0200
+++ src/java/org/apache/hadoop/mapred/pipes/Submitter.java 2008-07-23 13:01:58.000000000 +0200
@@ -364,6 +364,14 @@
}
if (results.hasOption("-jar")) {
conf.setJar((String) results.getValue("-jar"));
+ // if they gave us a jar file, include it into the class path
+ String jarFile = conf.getJar();
+ if (jarFile != null) {
+ ClassLoader loader =
+ new URLClassLoader(new URL[]{ FileSystem.getLocal(conf).
+ pathToFile(new Path(jarFile)).toURL()});
+ conf.setClassLoader(loader);
+ }
}
if (results.hasOption("-inputformat")) {
setIsJavaRecordReader(conf, true);
@@ -406,14 +414,6 @@
conf.set(keyValSplit[0], keyValSplit[1]);
}
}
- // if they gave us a jar file, include it into the class path
- String jarFile = conf.getJar();
- if (jarFile != null) {
- ClassLoader loader =
- new URLClassLoader(new URL[]{ FileSystem.getLocal(conf).
- pathToFile(new Path(jarFile)).toURL()});
- conf.setClassLoader(loader);
- }
submitJob(conf);
} catch (OptionException oe) {
cli.printUsage();
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4020) Hadoop Pipes do not load custom
InputFormats at appropriate time, rendering them useless in certain
scenarios
Posted by "Mateusz Berezecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mateusz Berezecki updated HADOOP-4020:
--------------------------------------
Description:
There's a patch provided below.
It was created on the 0.17.0 version,
but with a little tweaking can be applied
to the 0.18.0 as well. It fixes a timing bug,
whereas a custom inputformat was loaded
from the jar file, but the inputformat was being
processed much earlier in time, resulting
in a class not found exception.
was:
There's a patch provided below.
It was created on the 0.17.0 version,
but with a little tweaking can be applied
to the 0.18.0 as well. It fixes a timing bug,
whereas a custom inputformat was loaded
from the jar file, but the inputformat was being
processed much earlier in time, resulting
in a class not found exception.
--- src/java/org/apache/hadoop/mapred/pipes/Submitter.java.orig 2008-05-15 09:20:16.000000000 +0200
+++ src/java/org/apache/hadoop/mapred/pipes/Submitter.java 2008-07-23 13:01:58.000000000 +0200
@@ -364,6 +364,14 @@
}
if (results.hasOption("-jar")) {
conf.setJar((String) results.getValue("-jar"));
+ // if they gave us a jar file, include it into the class path
+ String jarFile = conf.getJar();
+ if (jarFile != null) {
+ ClassLoader loader =
+ new URLClassLoader(new URL[]{ FileSystem.getLocal(conf).
+ pathToFile(new Path(jarFile)).toURL()});
+ conf.setClassLoader(loader);
+ }
}
if (results.hasOption("-inputformat")) {
setIsJavaRecordReader(conf, true);
@@ -406,14 +414,6 @@
conf.set(keyValSplit[0], keyValSplit[1]);
}
}
- // if they gave us a jar file, include it into the class path
- String jarFile = conf.getJar();
- if (jarFile != null) {
- ClassLoader loader =
- new URLClassLoader(new URL[]{ FileSystem.getLocal(conf).
- pathToFile(new Path(jarFile)).toURL()});
- conf.setClassLoader(loader);
- }
submitJob(conf);
} catch (OptionException oe) {
cli.printUsage();
> Hadoop Pipes do not load custom InputFormats at appropriate time, rendering them useless in certain scenarios
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4020
> URL: https://issues.apache.org/jira/browse/HADOOP-4020
> Project: Hadoop Core
> Issue Type: Improvement
> Components: pipes
> Affects Versions: 0.16.4, 0.17.0, 0.17.1, 0.17.2
> Reporter: Mateusz Berezecki
>
> There's a patch provided below.
> It was created on the 0.17.0 version,
> but with a little tweaking can be applied
> to the 0.18.0 as well. It fixes a timing bug,
> whereas a custom inputformat was loaded
> from the jar file, but the inputformat was being
> processed much earlier in time, resulting
> in a class not found exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4020) Hadoop Pipes do not load custom
InputFormats at appropriate time, rendering them useless in certain
scenarios
Posted by "Mateusz Berezecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mateusz Berezecki updated HADOOP-4020:
--------------------------------------
Attachment: custom_format.patch
> Hadoop Pipes do not load custom InputFormats at appropriate time, rendering them useless in certain scenarios
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-4020
> URL: https://issues.apache.org/jira/browse/HADOOP-4020
> Project: Hadoop Core
> Issue Type: Improvement
> Components: pipes
> Affects Versions: 0.16.4, 0.17.0, 0.17.1, 0.17.2
> Reporter: Mateusz Berezecki
> Attachments: custom_format.patch
>
>
> There's a patch provided below.
> It was created on the 0.17.0 version,
> but with a little tweaking can be applied
> to the 0.18.0 as well. It fixes a timing bug,
> whereas a custom inputformat was loaded
> from the jar file, but the inputformat was being
> processed much earlier in time, resulting
> in a class not found exception.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.