You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@asterixdb.apache.org by Pääkkönen Pekka <Pe...@vtt.fi> on 2015/11/03 07:26:39 UTC

Re: AsterixDB - UDF

Hi,

Yes, I have been testing UDF in AQL.
It seems that I get a throughput of ~200 queries/s, when using multithreading in the client.
By batching 100 UDF operations/query, throughput increases to +10000 UDF operations/s.

I am wondering, if UDF is connected to a feed, how can I check that the UDF is processed fast enough, if
there would be real time processing requirements?

Regards,
-Pekka







You can test UDFs (and should!!) by using them in AQL queries against
the kinds of data that will be coming in on the feed.  Once they are
tested and working 100%, you can use them on a feed and be confident
that they'll be doing the right thing. Does that help?

Cheers,
Mike


On 10/28/15 3:53 AM, Pääkkönen Pekka wrote:
>
> Hi,
>
> Thanks. I am now able to create a simple UDF.
>
> If I connect the UDF to a feed, is it possible to know from outside of
> AsterixDB, when execution of UDF calls have been completed?
>
> Or should I use AQL via REST API (to call the UDF) for such purpose?
>
> -Pekka
>
> Hi Pekka,
>
> For making UDFs, you should not necessarily have to import AsterixDB
>
> into Eclipse/Intellij, as long as you are comfortable with maven and
>
> your text editor of choice. It may be easiest however.
>
> As for the content of the asterix-external-data zip, yes, other UDFs
>
> should follow the general example of that archive. That XML file (
>
> https://github.com/apache/incubator-asterixdb/blob/master/asterix-external-data/src/test/resources/library_descriptor.xml)
>
> corresponds to these set of functions:
>
> (https://github.com/apache/incubator-asterixdb/tree/master/asterix-external-data/src/test/java/org/apache/asterix/external/library)
>
> Hope that helps! Sorry there isn't a more end-to-end documentation
>
> regarding implementing a UDF, but hopefully the source examples are
>
> illustrative enough.
>
> Thanks,
>
> -Ian
>
> On Mon, Oct 26, 2015 at 6:02 AM, Pääkkönen Pekka
> <Pe...@vtt.fi> wrote:
>
> > Hi,
>
> >
>
> >
>
> >
>
> > In order to program UDFs, do I need to install AsterixDB development
>
> > environment to Eclipse?
>
> >
>
> >
>
> >
>
> > I am also wondering about the content of the UDF ZIP bundle.
>
> >
>
> > Should the content be similar to
>
> > asterix-external-data-0.8.7-SNAPSHOT-binary-assembly.zip, which contains
>
> >
>
> > a JAR and library_descriptor.xml?
>
> >
>
> >
>
> >
>
> > Regards,
>
> >
>
> > -Pekka
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>


Re: AsterixDB - UDF

Posted by Michael Carey <mj...@ics.uci.edu>.
Good to hear!  That's a really good question about "fast enough" - 
conversely, a customer might ask "so how much hardware will I need to 
buy in order to keep up with my ingestion requirements?".  Hmmm.  My 
thought is that you could take one node of your cluster and throw data 
at it as fast as you can from another node - do so with the feed set to 
use the "discard" policy and configured with your UDF - run it for 
awhile - then run a count query at the end to see how many data items 
make it to disk - and finally, divide that number by the length of your 
experiment.  (Do run it for awhile, say 1/2 hour.)  That will give you 
the processing capacity of one node.  The number of nodes you'll need 
for your workload can then be computed by looking at your desired (max) 
data arrival rate and dividing that by the result of your experiment - 
assuming linear scaling, which is roughly true for feeds (at least based 
on the experiments Raman Grover, aka FeedMan, did in his PhD work :-)).

Cheers,
Mike


On 11/2/15 10:26 PM, Pääkkönen Pekka wrote:
>
> Hi,
>
> Yes, I have been testing UDF in AQL.
>
> It seems that I get a throughput of ~200 queries/s, when using 
> multithreading in the client.
>
> By batching 100 UDF operations/query, throughput increases to +10000 
> UDF operations/s.
>
> I am wondering, if UDF is connected to a feed, how can I check that 
> the UDF is processed fast enough, if
>
> there would be real time processing requirements?
>
> Regards,
>
> -Pekka
>
> You can test UDFs (and should!!) by using them in AQL queries against
>
> the kinds of data that will be coming in on the feed.  Once they are
>
> tested and working 100%, you can use them on a feed and be confident
>
> that they'll be doing the right thing. Does that help?
>
> Cheers,
>
> Mike
>
> On 10/28/15 3:53 AM, Pääkkönen Pekka wrote:
>
> >
>
> > Hi,
>
> >
>
> > Thanks. I am now able to create a simple UDF.
>
> >
>
> > If I connect the UDF to a feed, is it possible to know from outside of
>
> > AsterixDB, when execution of UDF calls have been completed?
>
> >
>
> > Or should I use AQL via REST API (to call the UDF) for such purpose?
>
> >
>
> > -Pekka
>
> >
>
> > Hi Pekka,
>
> >
>
> > For making UDFs, you should not necessarily have to import AsterixDB
>
> >
>
> > into Eclipse/Intellij, as long as you are comfortable with maven and
>
> >
>
> > your text editor of choice. It may be easiest however.
>
> >
>
> > As for the content of the asterix-external-data zip, yes, other UDFs
>
> >
>
> > should follow the general example of that archive. That XML file (
>
> >
>
> > https://github.com/apache/incubator-asterixdb/blob/master/asterix-external-data/src/test/resources/library_descriptor.xml)
>
> >
>
> > corresponds to these set of functions:
>
> >
>
> > (https://github.com/apache/incubator-asterixdb/tree/master/asterix-external-data/src/test/java/org/apache/asterix/external/library)
>
> >
>
> > Hope that helps! Sorry there isn't a more end-to-end documentation
>
> >
>
> > regarding implementing a UDF, but hopefully the source examples are
>
> >
>
> > illustrative enough.
>
> >
>
> > Thanks,
>
> >
>
> > -Ian
>
> >
>
> > On Mon, Oct 26, 2015 at 6:02 AM, Pääkkönen Pekka
>
> > <Pe...@vtt.fi> wrote:
>
> >
>
> > > Hi,
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > In order to program UDFs, do I need to install AsterixDB development
>
> >
>
> > > environment to Eclipse?
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > I am also wondering about the content of the UDF ZIP bundle.
>
> >
>
> > >
>
> >
>
> > > Should the content be similar to
>
> >
>
> > > asterix-external-data-0.8.7-SNAPSHOT-binary-assembly.zip, which contains
>
> >
>
> > >
>
> >
>
> > > a JAR and library_descriptor.xml?
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > > Regards,
>
> >
>
> > >
>
> >
>
> > > -Pekka
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>