You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Dennis Kubes <nu...@dragonflymc.com> on 2006/08/29 17:02:10 UTC
Number of Reduce Outputs
This is probably a simple question but when I run my MR job I am getting
10 splits and therefore 10 output files like part-xxxxx. Is there a way
to merge those outputs into a single file using the currently running MR
job or do I need to run another MR job to merge them?
Dennis Kubes
Re: Number of Reduce Outputs
Posted by Frédéric Bertin <fr...@anyware-tech.com>.
I asked me the same question when I stepped into Hadoop, and I think
it's a good candidate for FAQ ;)
Generally speaking, IMO there is a need in Hadoop (MapReduce part) for
some kind of JobListener interface, allowing to write custom callbacks
called at strategic moments of a Job's life, and executed on a single
machine.
Dennis's problem could then be solved using a MergeOutputFilesListener.
This could also allow to do more complex things like notifying people of
jobs' results by mail, etc... but this kind of example may be outside
Hadoop's scope. However just publishing the listener interface would
contribute to make Hadoop more pluggable, and allow people to contribute
useful extensions, even if they are not focused on Hadoop's core.
WDYT?
Fred
Doug Cutting wrote:
> To generate a single output file, specify just a single reduce task.
> If your reducer isn't doing much computation, then it might be faster
> to do this in the original job, otherwise use a subsequent job.
>
> Doug
>
> Dennis Kubes wrote:
>> This is probably a simple question but when I run my MR job I am
>> getting 10 splits and therefore 10 output files like part-xxxxx. Is
>> there a way to merge those outputs into a single file using the
>> currently running MR job or do I need to run another MR job to merge
>> them?
>>
>> Dennis Kubes
Re: Number of Reduce Outputs
Posted by Doug Cutting <cu...@apache.org>.
To generate a single output file, specify just a single reduce task. If
your reducer isn't doing much computation, then it might be faster to do
this in the original job, otherwise use a subsequent job.
Doug
Dennis Kubes wrote:
> This is probably a simple question but when I run my MR job I am getting
> 10 splits and therefore 10 output files like part-xxxxx. Is there a way
> to merge those outputs into a single file using the currently running MR
> job or do I need to run another MR job to merge them?
>
> Dennis Kubes
Re: mapred.JobTracker : Connection refused
Posted by Dennis Kubes <nu...@dragonflymc.com>.
One of the servers probably didn't shut down completely. You would need
to check each of your nodes like this:
ps -ef | grep java
See if any of the servers are running. If they are they can be killed
manually like this where you put in the process_id
kill -9 process_id
Dennis
Sanjay Singh wrote:
> Hi,
>
> My nutch 0.8 intallation is 1 week old and has
> been pretty stable. Today I was doing some
> house-keeping, removing directories etc. Then I run
> the stop-all/start-all and I see this exception in the
> hadoop-nutch-jobtracker-data.log. It is constantly
> spewing out this exception, and I have a broken system
> :(
>
> I have tried applying the Lease patch
> (DFSClient.java (revision 423872)
> ) and have also upgraded to 0.4.1 with no solution.
>
>
> ======================================================
>
> 2006-09-10 19:40:13,483 WARN mapred.JobTracker
> (JobTracker.java:startTracker(68)) - Starting tracker
> java.net.ConnectException: Connection refused
> at
> java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> at
> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
> at
> java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> at
> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:507)
> at java.net.Socket.connect(Socket.java:457)
> at java.net.Socket.<init>(Socket.java:365)
> at java.net.Socket.<init>(Socket.java:207)
> at
> org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:113)
> at
> org.apache.hadoop.ipc.Client.getConnection(Client.java:359)
> at
> org.apache.hadoop.ipc.Client.call(Client.java:297)
> at
> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:150)
> at org.apache.hadoop.dfs.$Proxy0.isDir(Unknown
> Source)
> at
> org.apache.hadoop.dfs.DFSClient.isDirectory(DFSClient.java:325)
> at
> org.apache.hadoop.dfs.DistributedFileSystem.isDirectory(DistributedFileSystem.java:135)
> at
> org.apache.hadoop.fs.FileSystem.delete(FileSystem.java:414)
> at
> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:430)
> at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:65)
> at
> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:1133)
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
mapred.JobTracker : Connection refused
Posted by Sanjay Singh <ss...@yahoo.com>.
Hi,
My nutch 0.8 intallation is 1 week old and has
been pretty stable. Today I was doing some
house-keeping, removing directories etc. Then I run
the stop-all/start-all and I see this exception in the
hadoop-nutch-jobtracker-data.log. It is constantly
spewing out this exception, and I have a broken system
:(
I have tried applying the Lease patch
(DFSClient.java (revision 423872)
) and have also upgraded to 0.4.1 with no solution.
======================================================
2006-09-10 19:40:13,483 WARN mapred.JobTracker
(JobTracker.java:startTracker(68)) - Starting tracker
java.net.ConnectException: Connection refused
at
java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at
java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:507)
at java.net.Socket.connect(Socket.java:457)
at java.net.Socket.<init>(Socket.java:365)
at java.net.Socket.<init>(Socket.java:207)
at
org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:113)
at
org.apache.hadoop.ipc.Client.getConnection(Client.java:359)
at
org.apache.hadoop.ipc.Client.call(Client.java:297)
at
org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:150)
at org.apache.hadoop.dfs.$Proxy0.isDir(Unknown
Source)
at
org.apache.hadoop.dfs.DFSClient.isDirectory(DFSClient.java:325)
at
org.apache.hadoop.dfs.DistributedFileSystem.isDirectory(DistributedFileSystem.java:135)
at
org.apache.hadoop.fs.FileSystem.delete(FileSystem.java:414)
at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:430)
at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:65)
at
org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:1133)
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com