MapReduce causing output_xml.properties exception

[Steven Willis <on...@gmail.com>]

There was a question posed to this list from Pooya Woodcock on July 11
(http://mail-archives.apache.org/mod_mbox/hbase-user/201307.mbox/%3C20130711182941.GA27951@malkovich.internal.integralnet%3E)
that was apparently never answered. I've been having the exact same
issue with plain mapreduce and may have figured it out, so I thought
I'd post here in case anyone is looking for the same answer down the
line.

TLDR: restart all your hadoop services, the JVM may have been updated
recently without the services being restarted.

This is the output I (and Pooya) get from a dead simple MR job:

com.sun.org.apache.xml.internal.serializer.utils.WrappedRuntimeException:
Could not load the propery file 'output_xml.properties' for output
method 'xml' (check CLASSPATH)
    at com.sun.org.apache.xml.internal.serializer.OutputPropertiesFactory.getDefaultMethodProperties(OutputPropertiesFactory.java:344)
    at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.setDefaults(TransformerImpl.java:1149)
    at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.createOutputProperties(TransformerImpl.java:1110)
    at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.<init>(TransformerImpl.java:258)
    at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.<init>(TransformerImpl.java:249)
    at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:646)
    at org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:1270)
    at org.apache.hadoop.conf.Configuration.writeXml(Configuration.java:1255)
    at org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:1806)
    at org.apache.hadoop.mapred.JobInProgress$3.run(JobInProgress.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
    at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:677)
    at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4031)
    at org.apache.hadoop.mapred.JobInitializationPoller$JobInitializationThread.initializeJobs(JobInitializationPoller.java:137)
    at org.apache.hadoop.mapred.JobInitializationPoller$JobInitializationThread.run(JobInitializationPoller.java:108)

The only other reference to this error was here:
http://xalan.apache.org/xalan-j/faq.html#faq-25 however this was no
help as I didn't have xalan installed separately from the version
included with the JVM. I can see that 'output_xml.properties' is
inside '/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.25.x86_64/jre/lib/resources.jar'
and I wrote a simple class to open it as a resource and also to call
the method that was failing from the stack trace and it ran just fine
on all nodes in my cluster:

import java.util.Properties;
import com.sun.org.apache.xml.internal.serializer.Method;
import com.sun.org.apache.xml.internal.serializer.OutputPropertiesFactory;

public class T {
    public static void main(String[] args) {
        System.out.println(OutputPropertiesFactory.class.getResource("output_xml.properties"));
        Properties p =
OutputPropertiesFactory.getDefaultMethodProperties(Method.XML);
    }
}

I then thought, we updated these system relatively recently... maybe
the jvm got updated and the hadoop services didn't get restarted and
now they're having issues finding stuff in the JVM jars. It looks like
that may have been it. Restarting all my hadoop services fixed the
problem.

-Steven Willis