Lock manager

[Andrzej Bialecki <ab...@getopt.org>]

Hi,

I noticed that there are some additions being made to the 
JobSubmissionProtocol. While we're on this subject, I'd like to propose 
the following addition.

Currently there is no mechanism to coordinate concurrent access to 
specific resources across the cluster. This is apparent in Nutch in the 
protocol plugins, which need this "global lock" mechanism to control 
simultaneous access to remote hosts (at the moment the workaround is to 
execute just one task simultaneously to handle the locking inside a 
single JVM).

We could devise a mechanism based on files, residing on DFS, but it 
would be too heavyweight - in the above case it would mean 
creating/reading/deleting millions of files per job.

I propose to add a simple locking mechanism to JobTracker, which would 
hold locks specific to a given job. Then each task executing on a 
tasktracker could send requests  to JobTracker to acquire/release such 
locks (identified by strings, perhaps with some grouping mechanism), 
with the usual semantics. The cleanup in case of crashed jobs would be 
simple - whenever a job is finished, JobTracker would remove all locks 
related to that job.

In this schema it would be the JobTracker node that runs this lock 
manager, but I can imagine that we could designate any of the 
tasktracker nodes to run this "lock manager task" for each job, if it 
ever becomes a scalability problem. We could also go even further and 
create two types of lock managers - global lock manager (for resources 
unique per cluster) and per-job lock managers (for resources unique per 
job).

Any comments?

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Re: Lock manager

[Doug Cutting <cu...@apache.org>]

Andrzej Bialecki wrote:
> Currently there is no mechanism to coordinate concurrent access to 
> specific resources across the cluster. This is apparent in Nutch in the 
> protocol plugins, which need this "global lock" mechanism to control 
> simultaneous access to remote hosts (at the moment the workaround is to 
> execute just one task simultaneously to handle the locking inside a 
> single JVM).

Central locking sounds like it could quickly become a bottleneck.

Currently what Nutch's fetcher does to observe politeness is to 
partition fetcher input by host, and to disable splitting of fetcher 
input files, so that all urls with a given host are always processed as 
a single task.  Since, for politeness, urls from a host must be 
processed serially, there's no advantage to doing it otherwise: if we 
permitted multiple nodes to synchronize their access to a host through a 
locking mechanism then fetching could go no faster.

So how is the current mechanism inadequate?  What are some use cases for 
centralized locking that cannot be solved by partitioning task input?

Doug