You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Edmon Begoli <eb...@gmail.com> on 2015/09/22 20:18:09 UTC
Executing Drill queries automatically (and reactively)
Has anyone tried, and is there a possible a facility to trigger Drill
queries to recalculate some aggregates statistics if the underlying data
has changed.
My scenario is pretty simple - I use Drill to query directories of files to
mine some statistics out.
Files are arriving at some reasonable frequency, once an hour or even once
a da
I want to implement an event handler that observes these file arriving
"events" and that triggers Drill queries.
Has anyone done anything like this already, and do you have any suggested
approaches, specially if they are Drill-specific.
Thank you,
Edmon
Re: Executing Drill queries automatically (and reactively)
Posted by Edmon Begoli <eb...@gmail.com>.
Probably, but have the file watcher as the triggering event for Drill.
Maybe using WatchService.
Should I perhaps propose a future Drill enhancement for this in JIRA?
On Tue, Sep 22, 2015 at 3:32 PM, Scott C. Cote <sc...@me.com> wrote:
> Sounds like an opportunity for drill triggers
>
> Sent from my iPhone
>
> > On Sep 22, 2015, at 2:18 PM, Edmon Begoli <eb...@gmail.com> wrote:
> >
> > Yes, that is my current thinking and most likely course of action.
> >
> >> On Tue, Sep 22, 2015 at 3:13 PM, Paul Ilechko <pi...@maprtech.com>
> wrote:
> >>
> >> Perhaps you could implement something external to Drill to watch the
> file
> >> directory (inotify?) and trigger a program that makes calls to Drill
> >> through JDBC/ODBC.
> >>
> >>> On Tue, Sep 22, 2015 at 2:18 PM, Edmon Begoli <eb...@gmail.com>
> wrote:
> >>>
> >>> Has anyone tried, and is there a possible a facility to trigger Drill
> >>> queries to recalculate some aggregates statistics if the underlying
> data
> >>> has changed.
> >>>
> >>> My scenario is pretty simple - I use Drill to query directories of
> files
> >> to
> >>> mine some statistics out.
> >>>
> >>> Files are arriving at some reasonable frequency, once an hour or even
> >> once
> >>> a da
> >>>
> >>> I want to implement an event handler that observes these file arriving
> >>> "events" and that triggers Drill queries.
> >>>
> >>> Has anyone done anything like this already, and do you have any
> suggested
> >>> approaches, specially if they are Drill-specific.
> >>>
> >>> Thank you,
> >>> Edmon
> >>
> >>
> >>
> >> --
> >> ----------------------------------
> >> Paul Ilechko
> >> Senior Systems Engineer
> >> MapR Technologies
> >> 908 331 2207
> >>
>
Re: Executing Drill queries automatically (and reactively)
Posted by "Scott C. Cote" <sc...@me.com>.
Sounds like an opportunity for drill triggers
Sent from my iPhone
> On Sep 22, 2015, at 2:18 PM, Edmon Begoli <eb...@gmail.com> wrote:
>
> Yes, that is my current thinking and most likely course of action.
>
>> On Tue, Sep 22, 2015 at 3:13 PM, Paul Ilechko <pi...@maprtech.com> wrote:
>>
>> Perhaps you could implement something external to Drill to watch the file
>> directory (inotify?) and trigger a program that makes calls to Drill
>> through JDBC/ODBC.
>>
>>> On Tue, Sep 22, 2015 at 2:18 PM, Edmon Begoli <eb...@gmail.com> wrote:
>>>
>>> Has anyone tried, and is there a possible a facility to trigger Drill
>>> queries to recalculate some aggregates statistics if the underlying data
>>> has changed.
>>>
>>> My scenario is pretty simple - I use Drill to query directories of files
>> to
>>> mine some statistics out.
>>>
>>> Files are arriving at some reasonable frequency, once an hour or even
>> once
>>> a da
>>>
>>> I want to implement an event handler that observes these file arriving
>>> "events" and that triggers Drill queries.
>>>
>>> Has anyone done anything like this already, and do you have any suggested
>>> approaches, specially if they are Drill-specific.
>>>
>>> Thank you,
>>> Edmon
>>
>>
>>
>> --
>> ----------------------------------
>> Paul Ilechko
>> Senior Systems Engineer
>> MapR Technologies
>> 908 331 2207
>>
Re: Executing Drill queries automatically (and reactively)
Posted by Edmon Begoli <eb...@gmail.com>.
Yes, that is my current thinking and most likely course of action.
On Tue, Sep 22, 2015 at 3:13 PM, Paul Ilechko <pi...@maprtech.com> wrote:
> Perhaps you could implement something external to Drill to watch the file
> directory (inotify?) and trigger a program that makes calls to Drill
> through JDBC/ODBC.
>
> On Tue, Sep 22, 2015 at 2:18 PM, Edmon Begoli <eb...@gmail.com> wrote:
>
> > Has anyone tried, and is there a possible a facility to trigger Drill
> > queries to recalculate some aggregates statistics if the underlying data
> > has changed.
> >
> > My scenario is pretty simple - I use Drill to query directories of files
> to
> > mine some statistics out.
> >
> > Files are arriving at some reasonable frequency, once an hour or even
> once
> > a da
> >
> > I want to implement an event handler that observes these file arriving
> > "events" and that triggers Drill queries.
> >
> > Has anyone done anything like this already, and do you have any suggested
> > approaches, specially if they are Drill-specific.
> >
> > Thank you,
> > Edmon
> >
>
>
>
> --
> ----------------------------------
> Paul Ilechko
> Senior Systems Engineer
> MapR Technologies
> 908 331 2207
>
Re: Executing Drill queries automatically (and reactively)
Posted by Paul Ilechko <pi...@maprtech.com>.
Perhaps you could implement something external to Drill to watch the file
directory (inotify?) and trigger a program that makes calls to Drill
through JDBC/ODBC.
On Tue, Sep 22, 2015 at 2:18 PM, Edmon Begoli <eb...@gmail.com> wrote:
> Has anyone tried, and is there a possible a facility to trigger Drill
> queries to recalculate some aggregates statistics if the underlying data
> has changed.
>
> My scenario is pretty simple - I use Drill to query directories of files to
> mine some statistics out.
>
> Files are arriving at some reasonable frequency, once an hour or even once
> a da
>
> I want to implement an event handler that observes these file arriving
> "events" and that triggers Drill queries.
>
> Has anyone done anything like this already, and do you have any suggested
> approaches, specially if they are Drill-specific.
>
> Thank you,
> Edmon
>
--
----------------------------------
Paul Ilechko
Senior Systems Engineer
MapR Technologies
908 331 2207