You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Jon Brisbin <jo...@jbrisbin.com> on 2010/08/20 20:26:18 UTC
Scala view server?
I'm looking at the scouchdb Scala view server and trying to write my own version that doesn't compile arbitrary Scala code but simply links to existing, pre-compiled classes. I have a question about the way view servers work, though:
Does each view create its own command process (and thus JVM)?
I started down a garden path of using Akka remote actors to link the view server calls to objects that actually perform the work but I'm noticing that an "add_fun" call doesn't say what view its associated with, so I have to assume that each view instantiates its own JVM, helper objects, etc...
Does it call the command-line I give it (right now a Bash script that calls my Scala class) for every request of the view or does it reuse processes already started (like pooling)?
The more I get into this, the less feasible it seems to tie some JVM-based view language to Couch and still keep it performant.
What about the Erlang to Java bridge (JInterface)? Could this be used instead of the command-line style view server to hook into a long-running JVM process (like Akka remote Actors)?
Thanks!
J. Brisbin
http://jbrisbin.com/
Re: Scala view server?
Posted by J Chris Anderson <jc...@apache.org>.
On Aug 21, 2010, at 6:08 AM, Jon Brisbin wrote:
>
> On Aug 20, 2010, at 6:21 PM, Kenny Stone wrote:
>
>> Couldn't you leverage the hovercraft API with JInterface pretty easily?
>
> Wasn't aware of it but read through the README. It seems to be missing some pretty important functionality (like views).
>
you shouldn't use Hovercraft unless you've already written your application in Erlang using the REST API and need to make very specific changes to your interactions with the database.
most people will be best off pretending Hovercraft doesn't exist.
>>
>> I'm not sure why you would want to use the JVM when you have a perfectly
>> capable erlang vm. Could you explain your use case to me? I'm curious.
>
>
> Because I'm not an Erlang programmer. I'm not much of a Scala programmer, either, but I know Java well enough to do fairly intricate things.
>
> I could probably learn to leverage Erlang but I can't learn a new language for every tool I want to use. It doesn't make a lot of sense to me to write all the application layers in different languages. That makes it very, very difficult for the person(s) coming after me.
>
> Thanks!
>
> J. Brisbin
> http://jbrisbin.com/
>
>
>
>
>
>
Re: Scala view server?
Posted by Jon Brisbin <jo...@jbrisbin.com>.
On Aug 20, 2010, at 6:21 PM, Kenny Stone wrote:
> Couldn't you leverage the hovercraft API with JInterface pretty easily?
Wasn't aware of it but read through the README. It seems to be missing some pretty important functionality (like views).
>
> I'm not sure why you would want to use the JVM when you have a perfectly
> capable erlang vm. Could you explain your use case to me? I'm curious.
Because I'm not an Erlang programmer. I'm not much of a Scala programmer, either, but I know Java well enough to do fairly intricate things.
I could probably learn to leverage Erlang but I can't learn a new language for every tool I want to use. It doesn't make a lot of sense to me to write all the application layers in different languages. That makes it very, very difficult for the person(s) coming after me.
Thanks!
J. Brisbin
http://jbrisbin.com/
Re: Scala view server?
Posted by Kenny Stone <ks...@connamara.com>.
Couldn't you leverage the hovercraft API with JInterface pretty easily?
I'm not sure why you would want to use the JVM when you have a perfectly
capable erlang vm. Could you explain your use case to me? I'm curious.
--
Kenny Stone
Connamara Systems, LLC
On Fri, Aug 20, 2010 at 2:15 PM, Paul Davis <pa...@gmail.com>wrote:
> On Fri, Aug 20, 2010 at 3:05 PM, Jon Brisbin <jo...@jbrisbin.com> wrote:
> >
> > On Aug 20, 2010, at 1:42 PM, Paul Davis wrote:
> >
> >> During a view build, a single process is used. Between two builds,
> >> different processes may be used. If two builds are occurring
> >> simultaneously, they will use two separate OS level processes. The
> >> add_fun will re-add the function each time it starts a view build. The
> >> reset command should remove references to installed functions.
> >>
> >> I think you could be right in that it'd be hard to integrate a JVM for
> >> view processes because of its memory footprint in some situations with
> >> lots of simultaneous view builds. I was going through and refactoring
> >> some of that code to behave a bit more nicely which would provide the
> >> ability for you to do something with JInterface semi easily (i've
> >> never worked with it directly, but the couch side would be easier than
> >> it is currently).
> >>
> >> No idea if/when I'll finish that work though. I hit a stumbling block
> >> with performance not improving much with the various optimizations I
> >> had so I have to go back and do some profiling to figure out what's
> >> going on.
> >>
> >> HTH,
> >> Paul Davis
> >
> >
> > In the <1 hour I've devoted to the topic so far, it seems like writing a
> native_query_server that delegates to a Java process that's communicating
> via JInterface would be the best way to integrate with Java/Scala. The Java
> code could dispatch to workers via a BlockingQueue.
> >
> > I'm not an Erlang whiz, so I'm using this as an excuse to learn a little
> Erlang (like I have time to learn yet another new language! :/) and see if I
> can do this myself...
> >
> > Thanks!
> >
> > J. Brisbin
> > http://jbrisbin.com/
> >
>
> Oh right, I forgot to suggest that. If you want to go ahead with
> trying the JInterface on the current trunk, your best bet is to write
> a file that matches the current Erlang view server implementation:
>
>
> http://github.com/apache/couchdb/blob/trunk/src/couchdb/couch_native_process.erl
>
> Feel free to ask questions if you get stuck.
>
Re: Scala view server?
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Aug 20, 2010 at 3:05 PM, Jon Brisbin <jo...@jbrisbin.com> wrote:
>
> On Aug 20, 2010, at 1:42 PM, Paul Davis wrote:
>
>> During a view build, a single process is used. Between two builds,
>> different processes may be used. If two builds are occurring
>> simultaneously, they will use two separate OS level processes. The
>> add_fun will re-add the function each time it starts a view build. The
>> reset command should remove references to installed functions.
>>
>> I think you could be right in that it'd be hard to integrate a JVM for
>> view processes because of its memory footprint in some situations with
>> lots of simultaneous view builds. I was going through and refactoring
>> some of that code to behave a bit more nicely which would provide the
>> ability for you to do something with JInterface semi easily (i've
>> never worked with it directly, but the couch side would be easier than
>> it is currently).
>>
>> No idea if/when I'll finish that work though. I hit a stumbling block
>> with performance not improving much with the various optimizations I
>> had so I have to go back and do some profiling to figure out what's
>> going on.
>>
>> HTH,
>> Paul Davis
>
>
> In the <1 hour I've devoted to the topic so far, it seems like writing a native_query_server that delegates to a Java process that's communicating via JInterface would be the best way to integrate with Java/Scala. The Java code could dispatch to workers via a BlockingQueue.
>
> I'm not an Erlang whiz, so I'm using this as an excuse to learn a little Erlang (like I have time to learn yet another new language! :/) and see if I can do this myself...
>
> Thanks!
>
> J. Brisbin
> http://jbrisbin.com/
>
Oh right, I forgot to suggest that. If you want to go ahead with
trying the JInterface on the current trunk, your best bet is to write
a file that matches the current Erlang view server implementation:
http://github.com/apache/couchdb/blob/trunk/src/couchdb/couch_native_process.erl
Feel free to ask questions if you get stuck.
Re: Scala view server?
Posted by Jon Brisbin <jo...@jbrisbin.com>.
On Aug 20, 2010, at 1:42 PM, Paul Davis wrote:
> During a view build, a single process is used. Between two builds,
> different processes may be used. If two builds are occurring
> simultaneously, they will use two separate OS level processes. The
> add_fun will re-add the function each time it starts a view build. The
> reset command should remove references to installed functions.
>
> I think you could be right in that it'd be hard to integrate a JVM for
> view processes because of its memory footprint in some situations with
> lots of simultaneous view builds. I was going through and refactoring
> some of that code to behave a bit more nicely which would provide the
> ability for you to do something with JInterface semi easily (i've
> never worked with it directly, but the couch side would be easier than
> it is currently).
>
> No idea if/when I'll finish that work though. I hit a stumbling block
> with performance not improving much with the various optimizations I
> had so I have to go back and do some profiling to figure out what's
> going on.
>
> HTH,
> Paul Davis
In the <1 hour I've devoted to the topic so far, it seems like writing a native_query_server that delegates to a Java process that's communicating via JInterface would be the best way to integrate with Java/Scala. The Java code could dispatch to workers via a BlockingQueue.
I'm not an Erlang whiz, so I'm using this as an excuse to learn a little Erlang (like I have time to learn yet another new language! :/) and see if I can do this myself...
Thanks!
J. Brisbin
http://jbrisbin.com/
Re: Scala view server?
Posted by Paul Davis <pa...@gmail.com>.
On Fri, Aug 20, 2010 at 2:26 PM, Jon Brisbin <jo...@jbrisbin.com> wrote:
> I'm looking at the scouchdb Scala view server and trying to write my own version that doesn't compile arbitrary Scala code but simply links to existing, pre-compiled classes. I have a question about the way view servers work, though:
>
> Does each view create its own command process (and thus JVM)?
>
> I started down a garden path of using Akka remote actors to link the view server calls to objects that actually perform the work but I'm noticing that an "add_fun" call doesn't say what view its associated with, so I have to assume that each view instantiates its own JVM, helper objects, etc...
>
> Does it call the command-line I give it (right now a Bash script that calls my Scala class) for every request of the view or does it reuse processes already started (like pooling)?
>
> The more I get into this, the less feasible it seems to tie some JVM-based view language to Couch and still keep it performant.
>
> What about the Erlang to Java bridge (JInterface)? Could this be used instead of the command-line style view server to hook into a long-running JVM process (like Akka remote Actors)?
>
> Thanks!
>
> J. Brisbin
> http://jbrisbin.com/
>
During a view build, a single process is used. Between two builds,
different processes may be used. If two builds are occurring
simultaneously, they will use two separate OS level processes. The
add_fun will re-add the function each time it starts a view build. The
reset command should remove references to installed functions.
I think you could be right in that it'd be hard to integrate a JVM for
view processes because of its memory footprint in some situations with
lots of simultaneous view builds. I was going through and refactoring
some of that code to behave a bit more nicely which would provide the
ability for you to do something with JInterface semi easily (i've
never worked with it directly, but the couch side would be easier than
it is currently).
No idea if/when I'll finish that work though. I hit a stumbling block
with performance not improving much with the various optimizations I
had so I have to go back and do some profiling to figure out what's
going on.
HTH,
Paul Davis