You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by paco <pa...@gmail.com> on 2015/10/06 23:07:14 UTC
WritableComparator implements Configurable
Hi,
I am trying to get the object Configuration (created from my main file)
from my custom object WritableComparator:
My custom object WritableComparator
public class KeyComparator extends WritableComparator implements
Configurable {
private Configuration conf;
public KeyComparator() {
super(KeyWritable.class, true);
int number = conf.getInt("number_intervals", -1); // Here I
have a conf=null
}
(...)
@Override
public Configuration getConf() {
return conf;
}
@Override
public void setConf(Configuration arg0) {
this.conf = arg0;
}
}
My main file is:
Job job = Job.getInstance(getConf());
Configuration config = job.getConfiguration();
(...)
config.setInt("number_intervals", 5);
job.setSortComparatorClass(KeyComparator.class);
job.waitForCompletion(true);
return 0;
Could somebody gives me a hand, please?
It it is not possible, do you know some other way to share information
between main file and WritableComparator. I am trying to do the same
issue with my WritableComparable too.
Many thanks
Re: WritableComparator implements Configurable
Posted by paco <pa...@gmail.com>.
I forgot putting the exception thrown:
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:886)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1001)
at
org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
... 12 more
Caused by: java.lang.NullPointerException
at com.letsdobigdata.KeyComparator.<init>(KeyComparator.java:27)
KeyComparator.java:27 is the line where I accessed to conf.whatever
Thanks.
On 06/10/15 23:07, paco wrote:
> Hi,
>
> I am trying to get the object Configuration (created from my main
> file) from my custom object WritableComparator:
>
> My custom object WritableComparator
>
> public class KeyComparator extends WritableComparator implements
> Configurable {
>
> private Configuration conf;
>
> public KeyComparator() {
> super(KeyWritable.class, true);
> int number = conf.getInt("number_intervals", -1); // Here I
> have a conf=null
>
> }
>
> (...)
>
> @Override
> public Configuration getConf() {
> return conf;
> }
>
> @Override
> public void setConf(Configuration arg0) {
> this.conf = arg0;
> }
>
> }
>
> My main file is:
>
> Job job = Job.getInstance(getConf());
> Configuration config = job.getConfiguration();
>
> (...)
> config.setInt("number_intervals", 5);
>
> job.setSortComparatorClass(KeyComparator.class);
>
> job.waitForCompletion(true);
>
> return 0;
>
>
> Could somebody gives me a hand, please?
>
> It it is not possible, do you know some other way to share information
> between main file and WritableComparator. I am trying to do the same
> issue with my WritableComparable too.
>
> Many thanks
>
Re: WritableComparator implements Configurable
Posted by paco <pa...@gmail.com>.
I forgot putting the exception thrown:
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:886)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1001)
at
org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
... 12 more
Caused by: java.lang.NullPointerException
at com.letsdobigdata.KeyComparator.<init>(KeyComparator.java:27)
KeyComparator.java:27 is the line where I accessed to conf.whatever
Thanks.
On 06/10/15 23:07, paco wrote:
> Hi,
>
> I am trying to get the object Configuration (created from my main
> file) from my custom object WritableComparator:
>
> My custom object WritableComparator
>
> public class KeyComparator extends WritableComparator implements
> Configurable {
>
> private Configuration conf;
>
> public KeyComparator() {
> super(KeyWritable.class, true);
> int number = conf.getInt("number_intervals", -1); // Here I
> have a conf=null
>
> }
>
> (...)
>
> @Override
> public Configuration getConf() {
> return conf;
> }
>
> @Override
> public void setConf(Configuration arg0) {
> this.conf = arg0;
> }
>
> }
>
> My main file is:
>
> Job job = Job.getInstance(getConf());
> Configuration config = job.getConfiguration();
>
> (...)
> config.setInt("number_intervals", 5);
>
> job.setSortComparatorClass(KeyComparator.class);
>
> job.waitForCompletion(true);
>
> return 0;
>
>
> Could somebody gives me a hand, please?
>
> It it is not possible, do you know some other way to share information
> between main file and WritableComparator. I am trying to do the same
> issue with my WritableComparable too.
>
> Many thanks
>
Re: WritableComparator implements Configurable
Posted by paco <pa...@gmail.com>.
I forgot putting the exception thrown:
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:886)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1001)
at
org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
... 12 more
Caused by: java.lang.NullPointerException
at com.letsdobigdata.KeyComparator.<init>(KeyComparator.java:27)
KeyComparator.java:27 is the line where I accessed to conf.whatever
Thanks.
On 06/10/15 23:07, paco wrote:
> Hi,
>
> I am trying to get the object Configuration (created from my main
> file) from my custom object WritableComparator:
>
> My custom object WritableComparator
>
> public class KeyComparator extends WritableComparator implements
> Configurable {
>
> private Configuration conf;
>
> public KeyComparator() {
> super(KeyWritable.class, true);
> int number = conf.getInt("number_intervals", -1); // Here I
> have a conf=null
>
> }
>
> (...)
>
> @Override
> public Configuration getConf() {
> return conf;
> }
>
> @Override
> public void setConf(Configuration arg0) {
> this.conf = arg0;
> }
>
> }
>
> My main file is:
>
> Job job = Job.getInstance(getConf());
> Configuration config = job.getConfiguration();
>
> (...)
> config.setInt("number_intervals", 5);
>
> job.setSortComparatorClass(KeyComparator.class);
>
> job.waitForCompletion(true);
>
> return 0;
>
>
> Could somebody gives me a hand, please?
>
> It it is not possible, do you know some other way to share information
> between main file and WritableComparator. I am trying to do the same
> issue with my WritableComparable too.
>
> Many thanks
>
Re: WritableComparator implements Configurable
Posted by paco <pa...@gmail.com>.
I forgot putting the exception thrown:
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:886)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1001)
at
org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:695)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
... 12 more
Caused by: java.lang.NullPointerException
at com.letsdobigdata.KeyComparator.<init>(KeyComparator.java:27)
KeyComparator.java:27 is the line where I accessed to conf.whatever
Thanks.
On 06/10/15 23:07, paco wrote:
> Hi,
>
> I am trying to get the object Configuration (created from my main
> file) from my custom object WritableComparator:
>
> My custom object WritableComparator
>
> public class KeyComparator extends WritableComparator implements
> Configurable {
>
> private Configuration conf;
>
> public KeyComparator() {
> super(KeyWritable.class, true);
> int number = conf.getInt("number_intervals", -1); // Here I
> have a conf=null
>
> }
>
> (...)
>
> @Override
> public Configuration getConf() {
> return conf;
> }
>
> @Override
> public void setConf(Configuration arg0) {
> this.conf = arg0;
> }
>
> }
>
> My main file is:
>
> Job job = Job.getInstance(getConf());
> Configuration config = job.getConfiguration();
>
> (...)
> config.setInt("number_intervals", 5);
>
> job.setSortComparatorClass(KeyComparator.class);
>
> job.waitForCompletion(true);
>
> return 0;
>
>
> Could somebody gives me a hand, please?
>
> It it is not possible, do you know some other way to share information
> between main file and WritableComparator. I am trying to do the same
> issue with my WritableComparable too.
>
> Many thanks
>