You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Ted Dunning <te...@gmail.com> on 2014/09/29 02:51:42 UTC
trying to write simple UDF
I am trying to write a simple UDF and am hitting some odd road-blocks.
The ultimate error that I get is this:
Query failed: Screen received stop request sent. Line 347, Column 44:
Unknown variable or type "table" [990f5e97-3faa-42b7-9557-1a9416fd6321]
My code is at https://gist.github.com/tdunning/22432450b9e27948b6b5
The basic idea is that I want to define a function with three arguments,
all strings.
One odd thing that I see and don't understand is that in the generated code
in the log, I see my code, but with NullableIntHolder as the type for my
variables:
279: //---- start of eval portion of hLookup function. ----//
280: complexWriter15 .setPosition((outIndex));
281: {
282:
org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter
writer = complexWriter15;
283: NullableIntHolder table = out4;
284: NullableIntHolder columns = out9;
285: NullableIntHolder key = out14;
286:
287: HbaseLookup$Lookup_eval: {
288: System.out.printf("key = %s\n", key.toString());
289: }
290:
291: }
292: if (!complexWriter15 .ok()) {
293: complexWriter15 .reset();
294: return false;
295: }
296: //---- end of eval portion of hLookup function. ----//
I am compiling Drill by using "mvn clean install -DskipTests" at the
top-level. Is there a special incarnation to remove old versions of my UDF?
Re: trying to write simple UDF
Posted by Ted Dunning <te...@gmail.com>.
Thanks. I went a little different on this and build a new module dependent
on drill-root. Compile and package time is now down to 36 seconds. I
haven't worked out the bugs in the process yet, but it seems promising.
On Sun, Sep 28, 2014 at 6:17 PM, Steven Phillips <sp...@maprtech.com>
wrote:
> If you are mainly focusing on testing a UDF, my recommendation would be to
> keep the UDF you are working on in its own maven project. Build a combined
> source/class jar, and copy it into the jars directory, replacing the old
> one each time. If you are rebuilding Drill, or even just java-exec each
> time you make a change, that will be too time consuming.
>
> On Sun, Sep 28, 2014 at 6:09 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > Ahhh... apparently much of the confusion was due to my complex compile
> test
> > steps. I was untarring into a slightly different directory than I
> thought.
> >
> > Never mind.
> >
> > Related to this, does anybody have any hints for fast edit/test cycles?
> > Currently, I am using 1980's level of methods of using mvn to compile and
> > sqlline to test. Seems antiquated. What is the better way in the
> presence
> > of code generation and such?
> >
> >
> > On Sun, Sep 28, 2014 at 6:03 PM, Ted Dunning <te...@gmail.com>
> > wrote:
> >
> > >
> > > For reference, my sqlline.log file is at
> > >
> > > https://gist.github.com/tdunning/c9509b3f3a69a99faeaf
> > >
> > > The query I was trying to run was
> > >
> > > SELECT hLookup(last_name, cast(first_name as varchar(30)),
> cast(full_name
> > > as varchar(30)))as x, full_name FROM cp.`employee.json` limit 2;
> > >
> > > This query:
> > >
> > > SELECT hLookup(last_name, first_name, full_name) as x, full_name FROM
> > > cp.`employee.json` limit 2;
> > >
> > > and many other minor variations produced exactly the same error.
> > >
> > >
> > >
> > > On Sun, Sep 28, 2014 at 5:51 PM, Ted Dunning <te...@gmail.com>
> > > wrote:
> > >
> > >>
> > >> I am trying to write a simple UDF and am hitting some odd road-blocks.
> > >>
> > >> The ultimate error that I get is this:
> > >>
> > >> Query failed: Screen received stop request sent. Line 347, Column 44:
> > >> Unknown variable or type "table"
> [990f5e97-3faa-42b7-9557-1a9416fd6321]
> > >>
> > >>
> > >> My code is at https://gist.github.com/tdunning/22432450b9e27948b6b5
> > >>
> > >> The basic idea is that I want to define a function with three
> arguments,
> > >> all strings.
> > >>
> > >> One odd thing that I see and don't understand is that in the generated
> > >> code in the log, I see my code, but with NullableIntHolder as the type
> > for
> > >> my variables:
> > >> 279: //---- start of eval portion of hLookup function.
> > >> ----//
> > >> 280: complexWriter15 .setPosition((outIndex));
> > >> 281: {
> > >> 282:
> > >> org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter
> > >> writer = complexWriter15;
> > >> 283: NullableIntHolder table = out4;
> > >> 284: NullableIntHolder columns = out9;
> > >> 285: NullableIntHolder key = out14;
> > >> 286:
> > >> 287: HbaseLookup$Lookup_eval: {
> > >> 288: System.out.printf("key = %s\n", key.toString());
> > >> 289: }
> > >> 290:
> > >> 291: }
> > >> 292: if (!complexWriter15 .ok()) {
> > >> 293: complexWriter15 .reset();
> > >> 294: return false;
> > >> 295: }
> > >> 296: //---- end of eval portion of hLookup function.
> > ----//
> > >>
> > >> I am compiling Drill by using "mvn clean install -DskipTests" at the
> > >> top-level. Is there a special incarnation to remove old versions of
> my
> > UDF?
> > >>
> > >>
> > >>
> > >
> >
>
>
>
> --
> Steven Phillips
> Software Engineer
>
> mapr.com
>
Re: trying to write simple UDF
Posted by Steven Phillips <sp...@maprtech.com>.
If you are mainly focusing on testing a UDF, my recommendation would be to
keep the UDF you are working on in its own maven project. Build a combined
source/class jar, and copy it into the jars directory, replacing the old
one each time. If you are rebuilding Drill, or even just java-exec each
time you make a change, that will be too time consuming.
On Sun, Sep 28, 2014 at 6:09 PM, Ted Dunning <te...@gmail.com> wrote:
> Ahhh... apparently much of the confusion was due to my complex compile test
> steps. I was untarring into a slightly different directory than I thought.
>
> Never mind.
>
> Related to this, does anybody have any hints for fast edit/test cycles?
> Currently, I am using 1980's level of methods of using mvn to compile and
> sqlline to test. Seems antiquated. What is the better way in the presence
> of code generation and such?
>
>
> On Sun, Sep 28, 2014 at 6:03 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> >
> > For reference, my sqlline.log file is at
> >
> > https://gist.github.com/tdunning/c9509b3f3a69a99faeaf
> >
> > The query I was trying to run was
> >
> > SELECT hLookup(last_name, cast(first_name as varchar(30)), cast(full_name
> > as varchar(30)))as x, full_name FROM cp.`employee.json` limit 2;
> >
> > This query:
> >
> > SELECT hLookup(last_name, first_name, full_name) as x, full_name FROM
> > cp.`employee.json` limit 2;
> >
> > and many other minor variations produced exactly the same error.
> >
> >
> >
> > On Sun, Sep 28, 2014 at 5:51 PM, Ted Dunning <te...@gmail.com>
> > wrote:
> >
> >>
> >> I am trying to write a simple UDF and am hitting some odd road-blocks.
> >>
> >> The ultimate error that I get is this:
> >>
> >> Query failed: Screen received stop request sent. Line 347, Column 44:
> >> Unknown variable or type "table" [990f5e97-3faa-42b7-9557-1a9416fd6321]
> >>
> >>
> >> My code is at https://gist.github.com/tdunning/22432450b9e27948b6b5
> >>
> >> The basic idea is that I want to define a function with three arguments,
> >> all strings.
> >>
> >> One odd thing that I see and don't understand is that in the generated
> >> code in the log, I see my code, but with NullableIntHolder as the type
> for
> >> my variables:
> >> 279: //---- start of eval portion of hLookup function.
> >> ----//
> >> 280: complexWriter15 .setPosition((outIndex));
> >> 281: {
> >> 282:
> >> org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter
> >> writer = complexWriter15;
> >> 283: NullableIntHolder table = out4;
> >> 284: NullableIntHolder columns = out9;
> >> 285: NullableIntHolder key = out14;
> >> 286:
> >> 287: HbaseLookup$Lookup_eval: {
> >> 288: System.out.printf("key = %s\n", key.toString());
> >> 289: }
> >> 290:
> >> 291: }
> >> 292: if (!complexWriter15 .ok()) {
> >> 293: complexWriter15 .reset();
> >> 294: return false;
> >> 295: }
> >> 296: //---- end of eval portion of hLookup function.
> ----//
> >>
> >> I am compiling Drill by using "mvn clean install -DskipTests" at the
> >> top-level. Is there a special incarnation to remove old versions of my
> UDF?
> >>
> >>
> >>
> >
>
--
Steven Phillips
Software Engineer
mapr.com
Re: trying to write simple UDF
Posted by Ted Dunning <te...@gmail.com>.
Ahhh... apparently much of the confusion was due to my complex compile test
steps. I was untarring into a slightly different directory than I thought.
Never mind.
Related to this, does anybody have any hints for fast edit/test cycles?
Currently, I am using 1980's level of methods of using mvn to compile and
sqlline to test. Seems antiquated. What is the better way in the presence
of code generation and such?
On Sun, Sep 28, 2014 at 6:03 PM, Ted Dunning <te...@gmail.com> wrote:
>
> For reference, my sqlline.log file is at
>
> https://gist.github.com/tdunning/c9509b3f3a69a99faeaf
>
> The query I was trying to run was
>
> SELECT hLookup(last_name, cast(first_name as varchar(30)), cast(full_name
> as varchar(30)))as x, full_name FROM cp.`employee.json` limit 2;
>
> This query:
>
> SELECT hLookup(last_name, first_name, full_name) as x, full_name FROM
> cp.`employee.json` limit 2;
>
> and many other minor variations produced exactly the same error.
>
>
>
> On Sun, Sep 28, 2014 at 5:51 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
>>
>> I am trying to write a simple UDF and am hitting some odd road-blocks.
>>
>> The ultimate error that I get is this:
>>
>> Query failed: Screen received stop request sent. Line 347, Column 44:
>> Unknown variable or type "table" [990f5e97-3faa-42b7-9557-1a9416fd6321]
>>
>>
>> My code is at https://gist.github.com/tdunning/22432450b9e27948b6b5
>>
>> The basic idea is that I want to define a function with three arguments,
>> all strings.
>>
>> One odd thing that I see and don't understand is that in the generated
>> code in the log, I see my code, but with NullableIntHolder as the type for
>> my variables:
>> 279: //---- start of eval portion of hLookup function.
>> ----//
>> 280: complexWriter15 .setPosition((outIndex));
>> 281: {
>> 282:
>> org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter
>> writer = complexWriter15;
>> 283: NullableIntHolder table = out4;
>> 284: NullableIntHolder columns = out9;
>> 285: NullableIntHolder key = out14;
>> 286:
>> 287: HbaseLookup$Lookup_eval: {
>> 288: System.out.printf("key = %s\n", key.toString());
>> 289: }
>> 290:
>> 291: }
>> 292: if (!complexWriter15 .ok()) {
>> 293: complexWriter15 .reset();
>> 294: return false;
>> 295: }
>> 296: //---- end of eval portion of hLookup function. ----//
>>
>> I am compiling Drill by using "mvn clean install -DskipTests" at the
>> top-level. Is there a special incarnation to remove old versions of my UDF?
>>
>>
>>
>
Re: trying to write simple UDF
Posted by Ted Dunning <te...@gmail.com>.
For reference, my sqlline.log file is at
https://gist.github.com/tdunning/c9509b3f3a69a99faeaf
The query I was trying to run was
SELECT hLookup(last_name, cast(first_name as varchar(30)), cast(full_name
as varchar(30)))as x, full_name FROM cp.`employee.json` limit 2;
This query:
SELECT hLookup(last_name, first_name, full_name) as x, full_name FROM
cp.`employee.json` limit 2;
and many other minor variations produced exactly the same error.
On Sun, Sep 28, 2014 at 5:51 PM, Ted Dunning <te...@gmail.com> wrote:
>
> I am trying to write a simple UDF and am hitting some odd road-blocks.
>
> The ultimate error that I get is this:
>
> Query failed: Screen received stop request sent. Line 347, Column 44:
> Unknown variable or type "table" [990f5e97-3faa-42b7-9557-1a9416fd6321]
>
>
> My code is at https://gist.github.com/tdunning/22432450b9e27948b6b5
>
> The basic idea is that I want to define a function with three arguments,
> all strings.
>
> One odd thing that I see and don't understand is that in the generated
> code in the log, I see my code, but with NullableIntHolder as the type for
> my variables:
> 279: //---- start of eval portion of hLookup function.
> ----//
> 280: complexWriter15 .setPosition((outIndex));
> 281: {
> 282:
> org.apache.drill.exec.vector.complex.writer.BaseWriter.ComplexWriter
> writer = complexWriter15;
> 283: NullableIntHolder table = out4;
> 284: NullableIntHolder columns = out9;
> 285: NullableIntHolder key = out14;
> 286:
> 287: HbaseLookup$Lookup_eval: {
> 288: System.out.printf("key = %s\n", key.toString());
> 289: }
> 290:
> 291: }
> 292: if (!complexWriter15 .ok()) {
> 293: complexWriter15 .reset();
> 294: return false;
> 295: }
> 296: //---- end of eval portion of hLookup function. ----//
>
> I am compiling Drill by using "mvn clean install -DskipTests" at the
> top-level. Is there a special incarnation to remove old versions of my UDF?
>
>
>