Erlang views and static data

[Paul Hirst <pa...@sophos.com>]

Hi,

I'm currently rewriting some of my views in Erlang because I've had trouble with a few large documents exploding up the javascript interepreter and that's a good excuse to switch to Erlang and gain some extra performance too.

I have one view which is rather complex and which contains 2 dictionaries (hashes, objects, whatever you want to call them) each of around 150 elements. These are used to remap some of the data from one form to another. Every 6 months or so I need to add a couple of new entries to these and in the past that has meant rebuilding the whole view which takes a few days. That's all ok but I'm wondering how best to deal with this now I'm reimplementing in Erlang.

Firsly I'm not sure of the efficiencies of doing something like this:

Fun({Doc}) ->
  Lookup1 = [{<<"foo">>,<<"bar">},{<<"baz">>,<<"wibble">>},....],
  %% Rest of the code which actually uses Lookup1
  OldVal = proplist:get_value(<<"to_remap">>, Doc),
  NewVal = proplist:get_value(OldVal, Lookup1),
  Emit(NewVal, something...)
end.

Will Lookup1 get recreated every time a document is processed by the view, or every time the view updater is kicked, or something else?

My example shows a proplist. It sounds like I should actually use a dict structure. It looks like I can pass in the same list but run it through dict:from_list/1. That's all fine but my questions about efficiency apply even more.

Should/Can I keep this data outside of the view code? I fully understand the need to detect changes which alter the view output but in this case I can actually cheat by promising to only ever add entries, and to always add entries to the dictionary prior to adding and documents which require the change. Obviously I would need to tread very carefully but I could read the data from a file somewhere on disk... or maybe even from another document in the database.

Thanks,
Paul

________________________________

Sophos Limited, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
Company Reg No 2096520. VAT Reg No GB 991 2418 08.

Re: Erlang views and static data

[Robert Newson <rn...@apache.org>]

Hi Paul,

Yes, Lookup1 will be created for every call to your function (once per updated document). There's no equivalent to commonjs includes here. You might get away with sticking it in the process dictionary but I doubt it takes long to create the variable anyway, perhaps measure your indexing speed before optimizing that?

Remember that the erlang view server has no sandbox. As long as you trust everyone that can update design documents, that's fine.

Finally, perhaps if you explained the problem that led you to this solution we might be able to provide further assistance.

B.

On 12 Jul 2012, at 09:12, Paul Hirst wrote:

> Hi,
> 
> I'm currently rewriting some of my views in Erlang because I've had trouble with a few large documents exploding up the javascript interepreter and that's a good excuse to switch to Erlang and gain some extra performance too.
> 
> I have one view which is rather complex and which contains 2 dictionaries (hashes, objects, whatever you want to call them) each of around 150 elements. These are used to remap some of the data from one form to another. Every 6 months or so I need to add a couple of new entries to these and in the past that has meant rebuilding the whole view which takes a few days. That's all ok but I'm wondering how best to deal with this now I'm reimplementing in Erlang.
> 
> Firsly I'm not sure of the efficiencies of doing something like this:
> 
> Fun({Doc}) ->
>  Lookup1 = [{<<"foo">>,<<"bar">},{<<"baz">>,<<"wibble">>},....],
>  %% Rest of the code which actually uses Lookup1
>  OldVal = proplist:get_value(<<"to_remap">>, Doc),
>  NewVal = proplist:get_value(OldVal, Lookup1),
>  Emit(NewVal, something...)
> end.
> 
> Will Lookup1 get recreated every time a document is processed by the view, or every time the view updater is kicked, or something else?
> 
> My example shows a proplist. It sounds like I should actually use a dict structure. It looks like I can pass in the same list but run it through dict:from_list/1. That's all fine but my questions about efficiency apply even more.
> 
> Should/Can I keep this data outside of the view code? I fully understand the need to detect changes which alter the view output but in this case I can actually cheat by promising to only ever add entries, and to always add entries to the dictionary prior to adding and documents which require the change. Obviously I would need to tread very carefully but I could read the data from a file somewhere on disk... or maybe even from another document in the database.
> 
> Thanks,
> Paul
> 
> ________________________________
> 
> Sophos Limited, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom.
> Company Reg No 2096520. VAT Reg No GB 991 2418 08.