Index list

[Mark Olliver <ma...@infectiousmedia.com>]

Hi All,

I am a little stuck on something and after advise on the best path. I have
a file which has within it a list of domain names, from that I can obtain a
unique list of the domains I have.
The harder part is I know need to index that list so that I end up with a
list of domain name,index. I have considered using a python UDF as I don't
think this is possible in pig directly but how do I then pass in the whole
list and retrieve back the whole list with indexes.
On top of that, once I have the new list I will store that on s3 (easy
bit). But next time I run the job I need to merge that list with the new
domains, keeping the old indexes and giving any new records and index
counting up from there. I expect here I could join the old and new unique
lists before passing to python, then in python know I only need to add an
index to the new records. I presume I can do the python part by effectively
just creating an array, but I guess my real issue is how I ensure this part
of my job is only handled as a reduce and not a map job, after which I
would then be running more map jobs.

Thanks

Mark

Mark Olliver
DevOps
InfectiousMedia