You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Stefán Baxter <st...@activitystream.com> on 2016/01/29 20:56:17 UTC
UDF to fetch fields from a Lucene Index - Does something similar exist
Hi,
We want to create and share a UDF that fetches additional query fields from
a Lucene index (via Complex writer).
I'm wondering if anything similar existed for Redis, Hbase or any other
data source. It would be greatly appreciated if someone could point me to a
meaningful starting point.
Regards,
-Stefan
Re: UDF to fetch fields from a Lucene Index - Does something similar exist
Posted by Jacques Nadeau <ja...@dremio.com>.
I happen to chat with Ted about this offline and I don't think he made it
all the way to working code.
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Sun, Jan 31, 2016 at 6:27 AM, Stefán Baxter <st...@activitystream.com>
wrote:
> Any takers?
>
> thanks for the pointers.
>
> -Stefan
>
> On Fri, Jan 29, 2016 at 9:11 PM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
> > I think maybe Ted Dunning built something against HBase leveraging this
> > technique. Ted, am I remembering right and if so, is that code somewhere
> > public?
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Fri, Jan 29, 2016 at 12:58 PM, Jason Altekruse <
> > altekrusejason@gmail.com>
> > wrote:
> >
> > > For a function that uses complex writer you could look at the
> > > convert_fromJSON [1] function, which takes a varchar containing JSON
> and
> > > parses it into a complex structure.
> > >
> > > The best example of reading from an external source would be the record
> > > readers for the actual format/storage plugins. There aren't really many
> > > limitations to what you can do from a UDF, so you can just put the
> > > initialization of your reader on the Lucene index in the setup method
> of
> > > the function and then read out if it as you would from a within a
> record
> > > reader.
> > >
> > > The JSON functions might be particularly useful, because it actually
> > shares
> > > the reader code between the record reader and the UDF, so it should be
> > able
> > > to demonstrate reasonable structure for sharing the code between the
> > lucene
> > > reader and the function.
> > >
> > > [1] -
> > >
> > >
> >
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/conv/JsonConvertFrom.java
> > >
> > > On Fri, Jan 29, 2016 at 11:56 AM, Stefán Baxter <
> > stefan@activitystream.com
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We want to create and share a UDF that fetches additional query
> fields
> > > from
> > > > a Lucene index (via Complex writer).
> > > >
> > > > I'm wondering if anything similar existed for Redis, Hbase or any
> other
> > > > data source. It would be greatly appreciated if someone could point
> me
> > > to a
> > > > meaningful starting point.
> > > >
> > > > Regards,
> > > > -Stefan
> > > >
> > >
> >
>
Re: UDF to fetch fields from a Lucene Index - Does something similar exist
Posted by Stefán Baxter <st...@activitystream.com>.
Any takers?
thanks for the pointers.
-Stefan
On Fri, Jan 29, 2016 at 9:11 PM, Jacques Nadeau <ja...@dremio.com> wrote:
> I think maybe Ted Dunning built something against HBase leveraging this
> technique. Ted, am I remembering right and if so, is that code somewhere
> public?
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Fri, Jan 29, 2016 at 12:58 PM, Jason Altekruse <
> altekrusejason@gmail.com>
> wrote:
>
> > For a function that uses complex writer you could look at the
> > convert_fromJSON [1] function, which takes a varchar containing JSON and
> > parses it into a complex structure.
> >
> > The best example of reading from an external source would be the record
> > readers for the actual format/storage plugins. There aren't really many
> > limitations to what you can do from a UDF, so you can just put the
> > initialization of your reader on the Lucene index in the setup method of
> > the function and then read out if it as you would from a within a record
> > reader.
> >
> > The JSON functions might be particularly useful, because it actually
> shares
> > the reader code between the record reader and the UDF, so it should be
> able
> > to demonstrate reasonable structure for sharing the code between the
> lucene
> > reader and the function.
> >
> > [1] -
> >
> >
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/conv/JsonConvertFrom.java
> >
> > On Fri, Jan 29, 2016 at 11:56 AM, Stefán Baxter <
> stefan@activitystream.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > We want to create and share a UDF that fetches additional query fields
> > from
> > > a Lucene index (via Complex writer).
> > >
> > > I'm wondering if anything similar existed for Redis, Hbase or any other
> > > data source. It would be greatly appreciated if someone could point me
> > to a
> > > meaningful starting point.
> > >
> > > Regards,
> > > -Stefan
> > >
> >
>
Re: UDF to fetch fields from a Lucene Index - Does something similar exist
Posted by Jacques Nadeau <ja...@dremio.com>.
I think maybe Ted Dunning built something against HBase leveraging this
technique. Ted, am I remembering right and if so, is that code somewhere
public?
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Fri, Jan 29, 2016 at 12:58 PM, Jason Altekruse <al...@gmail.com>
wrote:
> For a function that uses complex writer you could look at the
> convert_fromJSON [1] function, which takes a varchar containing JSON and
> parses it into a complex structure.
>
> The best example of reading from an external source would be the record
> readers for the actual format/storage plugins. There aren't really many
> limitations to what you can do from a UDF, so you can just put the
> initialization of your reader on the Lucene index in the setup method of
> the function and then read out if it as you would from a within a record
> reader.
>
> The JSON functions might be particularly useful, because it actually shares
> the reader code between the record reader and the UDF, so it should be able
> to demonstrate reasonable structure for sharing the code between the lucene
> reader and the function.
>
> [1] -
>
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/conv/JsonConvertFrom.java
>
> On Fri, Jan 29, 2016 at 11:56 AM, Stefán Baxter <stefan@activitystream.com
> >
> wrote:
>
> > Hi,
> >
> > We want to create and share a UDF that fetches additional query fields
> from
> > a Lucene index (via Complex writer).
> >
> > I'm wondering if anything similar existed for Redis, Hbase or any other
> > data source. It would be greatly appreciated if someone could point me
> to a
> > meaningful starting point.
> >
> > Regards,
> > -Stefan
> >
>
Re: UDF to fetch fields from a Lucene Index - Does something similar exist
Posted by Jason Altekruse <al...@gmail.com>.
For a function that uses complex writer you could look at the
convert_fromJSON [1] function, which takes a varchar containing JSON and
parses it into a complex structure.
The best example of reading from an external source would be the record
readers for the actual format/storage plugins. There aren't really many
limitations to what you can do from a UDF, so you can just put the
initialization of your reader on the Lucene index in the setup method of
the function and then read out if it as you would from a within a record
reader.
The JSON functions might be particularly useful, because it actually shares
the reader code between the record reader and the UDF, so it should be able
to demonstrate reasonable structure for sharing the code between the lucene
reader and the function.
[1] -
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/conv/JsonConvertFrom.java
On Fri, Jan 29, 2016 at 11:56 AM, Stefán Baxter <st...@activitystream.com>
wrote:
> Hi,
>
> We want to create and share a UDF that fetches additional query fields from
> a Lucene index (via Complex writer).
>
> I'm wondering if anything similar existed for Redis, Hbase or any other
> data source. It would be greatly appreciated if someone could point me to a
> meaningful starting point.
>
> Regards,
> -Stefan
>