You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Yash Sharma <ya...@impetus.co.in> on 2013/11/24 10:45:55 UTC
Type Casting
Observations while providing explicit cast functions.
1.
On hitting any explicit cast query on Sqlline I get the below results.
While debugging the flow I see that there is no lookup happening in FunctionImplementationRegistry for CAST() function, and therefore it is not casting the argument and returning the passed argument directly.
It seems like some part of query validation is bypassing the lookup for 'cast' function from FunctionRegistry.
0: jdbc:drill:schema=parquet-local> select cast(2.3 as int) from "sample-data/region.parquet";
+---------+
| EXPR$0 |
+---------+
| 2.3 |
| 2.3 |
| 2.3 |
| 2.3 |
| 2.3 |
+---------+
5 rows selected (0.489 seconds)
2.
For the 'Is Castable' check, we might have to create our own class similar to Optiq's SqlAssignmentRules. Since Drill's datatypes are different from those used in Optiq we might not be able to reuse the class directly.
(Ref: https://github.com/julianhyde/optiq/blob/master/core/src/main/java/org/eigenbase/sql/type/SqlTypeAssignmentRules.java)
Alternatively, an approach can be adopted from Hive's org.apache.hadoop.hive.ql.exec.FunctionRegistry's implicitConvertable() method. It uses datatype grouping via enum (PrimitiveGrouping). Then implicitlyConvertable() method checks the common groups to check if types are convertible implicitly.
Let me know your thoughts on the two approaches.
Regards,
Yash
________________________________________
From: Julian Hyde [julianhyde@gmail.com]
Sent: Friday, November 22, 2013 4:16 AM
To: drill-dev@incubator.apache.org
Subject: Re: hangout
On Nov 20, 2013, at 9:12 PM, Jinfeng Ni <ji...@gmail.com> wrote:
> 5. Nullable vs Non-nullable. I think implicit cast probably need
> support cast from Non-nullable exp
>
> Nullable exp. Otherwise, for each operator and each type, we at least
> have to implement 4 versions:
>
> 1) Nullable int + Nullable int
>
> 2) Nullable int + Non-Nullalbe int
>
> 3) Non-Nullable int + Nullable int
>
> 4) Non-Nullable int + Non-Nullalbe int.
>
> By implicit cast Non-nullable exp into Nullable exp, we only need
> define one version: Nullable int + Nullable int.
> . How to handle the null input could be specified in
> function/operator's implementation.
I faced this issue in Optiq's code generation. Since a lot of SQL operators generate null if any of their inputs are null, and since Optiq uses primitive types if it knows an expression cannot be null, I decided to implement only one version: Not-Nullable int + Not-Nullable int. If either argument is nullable, I add code around it to check for null values first.
Thus:
Integer x;
Integer y;
if (x == null) {
return null;
} else {
int x_ = x;
if (y == null) {
return null;
} else {
int y_ = y;
return x_ + y_; // this line is the only one produced by the implementor for '+'
}
}
You can see the gory details in RexToLixTranslator. In the "Expression translate(RexNode expr, RexImpTable.NullAs nullAs)" method, nullAs says whether null should cause the method to return null, true (for implementing "x is null"), false, or whether null is simply not possible (because we've already dealt with the possibility of nulls.
The supporting implementors (that do code generation for each operator or function) are in RexImpTable.
Julian
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Type Casting
Posted by Yash Sharma <ya...@impetus.co.in>.
Hi Jinfeng/All,
I have pushed the isCastable() check in my branch: https://github.com/yashs360/incubator-drill-casting
Let me know if it looks good, or any modifications are required. I will post a patch on it if everything works well.
Also, I will be travelling next week so my response might be lil late. Please keep me updated.
- Yash
________________________________________
From: Jinfeng Ni [jinfengni99@gmail.com]
Sent: Wednesday, November 27, 2013 12:00 AM
To: drill-dev@incubator.apache.org
Subject: Re: Type Casting
Hi Yash,
I completed a prototype of parsing part and explicit cast. So far, the
prototype works in the way I expect. Please continue your implicit cast
and isCastable check work. I'll keep you posted when I have any update.
On Tue, Nov 26, 2013 at 2:49 AM, Yash Sharma <ya...@impetus.co.in>wrote:
> Great Jinfeng,
> The parsing part is something I was stuck yesterday and was trying to
> figure out. Nice to know you are working on it.
> I would then continue with the isCastable check. Would implement it
> similar to Optiq's Map rather than Hive style.
> You can keep working on the parsing part. I would let you know when my
> part is done.
>
> - Yash
>
>
>
>
> -----Original Message-----
> From: Jinfeng Ni [mailto:jinfengni99@gmail.com]
> Sent: Monday, November 25, 2013 11:56 PM
> To: drill-dev@incubator.apache.org
> Subject: Re: Type Casting
>
> Hi Yash,
>
> For the checking of isCastable, I agree that we can not directly use
> Optiq's SqlAssignmentRules, since the date types are different. But looks
> like the implementation would be quite similar, as we could use Map<type,
> Set<type>> to check if one type is allowed to cast to another type.
>
> I'm working on the parser part to support explicit cast function, so that
> we can put cast function in logical/physical plan.
> How about you continue work on implicit cast and the check of isCastable?
> Once both the explicit cast and implicit cast is ready, we can discuss how
> to merge them together ( I assume implicit will eventually call the
> explicit cast function to do the real cast).
>
>
>
> On Sun, Nov 24, 2013 at 9:27 AM, Jacques Nadeau <ja...@apache.org>
> wrote:
>
> > For now, you need to do all cast work using logical or physical plans.
> > The Optiq currently removes any explicit casts from the logical
> > output. Once we depend on an updated Optiq with Mehant's ANY changes,
> > we can add explicit cast support back in.
> >
> > Jinfeng was working on some cast infrastructure as well. Jinfeng, can
> > you comment on the second part given the design brainstorming you've
> done?
> >
> > thanks,
> > Jacques
> >
> >
> > On Sun, Nov 24, 2013 at 1:45 AM, Yash Sharma
> > <yash.sharma@impetus.co.in
> > >wrote:
> >
> > > Observations while providing explicit cast functions.
> > >
> > > 1.
> > > On hitting any explicit cast query on Sqlline I get the below results.
> > > While debugging the flow I see that there is no lookup happening in
> > > FunctionImplementationRegistry for CAST() function, and therefore it
> > > is
> > not
> > > casting the argument and returning the passed argument directly.
> > > It seems like some part of query validation is bypassing the lookup
> > > for 'cast' function from FunctionRegistry.
> > >
> > > 0: jdbc:drill:schema=parquet-local> select cast(2.3 as int) from
> > > "sample-data/region.parquet";
> > > +---------+
> > > | EXPR$0 |
> > > +---------+
> > > | 2.3 |
> > > | 2.3 |
> > > | 2.3 |
> > > | 2.3 |
> > > | 2.3 |
> > > +---------+
> > > 5 rows selected (0.489 seconds)
> > >
> > >
> > >
> > > 2.
> > > For the 'Is Castable' check, we might have to create our own class
> > similar
> > > to Optiq's SqlAssignmentRules. Since Drill's datatypes are different
> > > from those used in Optiq we might not be able to reuse the class
> directly.
> > > (Ref:
> > >
> > https://github.com/julianhyde/optiq/blob/master/core/src/main/java/org
> > /eigenbase/sql/type/SqlTypeAssignmentRules.java
> > > )
> > >
> > > Alternatively, an approach can be adopted from Hive's
> > > org.apache.hadoop.hive.ql.exec.FunctionRegistry's
> > > implicitConvertable() method. It uses datatype grouping via enum
> > > (PrimitiveGrouping). Then
> > > implicitlyConvertable() method checks the common groups to check if
> > > types are convertible implicitly.
> > >
> > > Let me know your thoughts on the two approaches.
> > >
> > > Regards,
> > > Yash
> > >
> > >
> > >
> > >
> > > ________________________________________
> > > From: Julian Hyde [julianhyde@gmail.com]
> > > Sent: Friday, November 22, 2013 4:16 AM
> > > To: drill-dev@incubator.apache.org
> > > Subject: Re: hangout
> > >
> > > On Nov 20, 2013, at 9:12 PM, Jinfeng Ni <ji...@gmail.com> wrote:
> > >
> > > > 5. Nullable vs Non-nullable. I think implicit cast probably need
> > > > support cast from Non-nullable exp
> > > >
> > > > Nullable exp. Otherwise, for each operator and each type, we at
> > > > least have to implement 4 versions:
> > > >
> > > > 1) Nullable int + Nullable int
> > > >
> > > > 2) Nullable int + Non-Nullalbe int
> > > >
> > > > 3) Non-Nullable int + Nullable int
> > > >
> > > > 4) Non-Nullable int + Non-Nullalbe int.
> > > >
> > > > By implicit cast Non-nullable exp into Nullable exp, we only need
> > > > define one version: Nullable int + Nullable int.
> > > > . How to handle the null input could be specified in
> > > > function/operator's implementation.
> > >
> > > I faced this issue in Optiq's code generation. Since a lot of SQL
> > > operators generate null if any of their inputs are null, and since
> > > Optiq uses primitive types if it knows an expression cannot be null,
> > > I decided
> > to
> > > implement only one version: Not-Nullable int + Not-Nullable int. If
> > either
> > > argument is nullable, I add code around it to check for null values
> > first.
> > >
> > > Thus:
> > >
> > > Integer x;
> > > Integer y;
> > > if (x == null) {
> > > return null;
> > > } else {
> > > int x_ = x;
> > > if (y == null) {
> > > return null;
> > > } else {
> > > int y_ = y;
> > > return x_ + y_; // this line is the only one produced by the
> > > implementor for '+'
> > > }
> > > }
> > >
> > > You can see the gory details in RexToLixTranslator. In the
> > > "Expression translate(RexNode expr, RexImpTable.NullAs nullAs)"
> > > method, nullAs says whether null should cause the method to return
> > > null, true (for
> > implementing
> > > "x is null"), false, or whether null is simply not possible (because
> > we've
> > > already dealt with the possibility of nulls.
> > >
> > > The supporting implementors (that do code generation for each
> > > operator or
> > > function) are in RexImpTable.
> > >
> > > Julian
> > >
> > > ________________________________
> > >
> > >
> > >
> > >
> > >
> > >
> > > NOTE: This message may contain information that is confidential,
> > > proprietary, privileged or otherwise protected by law. The message
> > > is intended solely for the named addressee. If received in error,
> > > please destroy and notify the sender. Any use of this email is
> > > prohibited when received in error. Impetus does not represent,
> > > warrant and/or guarantee, that the integrity of this communication
> > > has been maintained nor that the communication is free of errors,
> virus, interception or interference.
> > >
> >
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Type Casting
Posted by Jinfeng Ni <ji...@gmail.com>.
Hi Yash,
I completed a prototype of parsing part and explicit cast. So far, the
prototype works in the way I expect. Please continue your implicit cast
and isCastable check work. I'll keep you posted when I have any update.
On Tue, Nov 26, 2013 at 2:49 AM, Yash Sharma <ya...@impetus.co.in>wrote:
> Great Jinfeng,
> The parsing part is something I was stuck yesterday and was trying to
> figure out. Nice to know you are working on it.
> I would then continue with the isCastable check. Would implement it
> similar to Optiq's Map rather than Hive style.
> You can keep working on the parsing part. I would let you know when my
> part is done.
>
> - Yash
>
>
>
>
> -----Original Message-----
> From: Jinfeng Ni [mailto:jinfengni99@gmail.com]
> Sent: Monday, November 25, 2013 11:56 PM
> To: drill-dev@incubator.apache.org
> Subject: Re: Type Casting
>
> Hi Yash,
>
> For the checking of isCastable, I agree that we can not directly use
> Optiq's SqlAssignmentRules, since the date types are different. But looks
> like the implementation would be quite similar, as we could use Map<type,
> Set<type>> to check if one type is allowed to cast to another type.
>
> I'm working on the parser part to support explicit cast function, so that
> we can put cast function in logical/physical plan.
> How about you continue work on implicit cast and the check of isCastable?
> Once both the explicit cast and implicit cast is ready, we can discuss how
> to merge them together ( I assume implicit will eventually call the
> explicit cast function to do the real cast).
>
>
>
> On Sun, Nov 24, 2013 at 9:27 AM, Jacques Nadeau <ja...@apache.org>
> wrote:
>
> > For now, you need to do all cast work using logical or physical plans.
> > The Optiq currently removes any explicit casts from the logical
> > output. Once we depend on an updated Optiq with Mehant's ANY changes,
> > we can add explicit cast support back in.
> >
> > Jinfeng was working on some cast infrastructure as well. Jinfeng, can
> > you comment on the second part given the design brainstorming you've
> done?
> >
> > thanks,
> > Jacques
> >
> >
> > On Sun, Nov 24, 2013 at 1:45 AM, Yash Sharma
> > <yash.sharma@impetus.co.in
> > >wrote:
> >
> > > Observations while providing explicit cast functions.
> > >
> > > 1.
> > > On hitting any explicit cast query on Sqlline I get the below results.
> > > While debugging the flow I see that there is no lookup happening in
> > > FunctionImplementationRegistry for CAST() function, and therefore it
> > > is
> > not
> > > casting the argument and returning the passed argument directly.
> > > It seems like some part of query validation is bypassing the lookup
> > > for 'cast' function from FunctionRegistry.
> > >
> > > 0: jdbc:drill:schema=parquet-local> select cast(2.3 as int) from
> > > "sample-data/region.parquet";
> > > +---------+
> > > | EXPR$0 |
> > > +---------+
> > > | 2.3 |
> > > | 2.3 |
> > > | 2.3 |
> > > | 2.3 |
> > > | 2.3 |
> > > +---------+
> > > 5 rows selected (0.489 seconds)
> > >
> > >
> > >
> > > 2.
> > > For the 'Is Castable' check, we might have to create our own class
> > similar
> > > to Optiq's SqlAssignmentRules. Since Drill's datatypes are different
> > > from those used in Optiq we might not be able to reuse the class
> directly.
> > > (Ref:
> > >
> > https://github.com/julianhyde/optiq/blob/master/core/src/main/java/org
> > /eigenbase/sql/type/SqlTypeAssignmentRules.java
> > > )
> > >
> > > Alternatively, an approach can be adopted from Hive's
> > > org.apache.hadoop.hive.ql.exec.FunctionRegistry's
> > > implicitConvertable() method. It uses datatype grouping via enum
> > > (PrimitiveGrouping). Then
> > > implicitlyConvertable() method checks the common groups to check if
> > > types are convertible implicitly.
> > >
> > > Let me know your thoughts on the two approaches.
> > >
> > > Regards,
> > > Yash
> > >
> > >
> > >
> > >
> > > ________________________________________
> > > From: Julian Hyde [julianhyde@gmail.com]
> > > Sent: Friday, November 22, 2013 4:16 AM
> > > To: drill-dev@incubator.apache.org
> > > Subject: Re: hangout
> > >
> > > On Nov 20, 2013, at 9:12 PM, Jinfeng Ni <ji...@gmail.com> wrote:
> > >
> > > > 5. Nullable vs Non-nullable. I think implicit cast probably need
> > > > support cast from Non-nullable exp
> > > >
> > > > Nullable exp. Otherwise, for each operator and each type, we at
> > > > least have to implement 4 versions:
> > > >
> > > > 1) Nullable int + Nullable int
> > > >
> > > > 2) Nullable int + Non-Nullalbe int
> > > >
> > > > 3) Non-Nullable int + Nullable int
> > > >
> > > > 4) Non-Nullable int + Non-Nullalbe int.
> > > >
> > > > By implicit cast Non-nullable exp into Nullable exp, we only need
> > > > define one version: Nullable int + Nullable int.
> > > > . How to handle the null input could be specified in
> > > > function/operator's implementation.
> > >
> > > I faced this issue in Optiq's code generation. Since a lot of SQL
> > > operators generate null if any of their inputs are null, and since
> > > Optiq uses primitive types if it knows an expression cannot be null,
> > > I decided
> > to
> > > implement only one version: Not-Nullable int + Not-Nullable int. If
> > either
> > > argument is nullable, I add code around it to check for null values
> > first.
> > >
> > > Thus:
> > >
> > > Integer x;
> > > Integer y;
> > > if (x == null) {
> > > return null;
> > > } else {
> > > int x_ = x;
> > > if (y == null) {
> > > return null;
> > > } else {
> > > int y_ = y;
> > > return x_ + y_; // this line is the only one produced by the
> > > implementor for '+'
> > > }
> > > }
> > >
> > > You can see the gory details in RexToLixTranslator. In the
> > > "Expression translate(RexNode expr, RexImpTable.NullAs nullAs)"
> > > method, nullAs says whether null should cause the method to return
> > > null, true (for
> > implementing
> > > "x is null"), false, or whether null is simply not possible (because
> > we've
> > > already dealt with the possibility of nulls.
> > >
> > > The supporting implementors (that do code generation for each
> > > operator or
> > > function) are in RexImpTable.
> > >
> > > Julian
> > >
> > > ________________________________
> > >
> > >
> > >
> > >
> > >
> > >
> > > NOTE: This message may contain information that is confidential,
> > > proprietary, privileged or otherwise protected by law. The message
> > > is intended solely for the named addressee. If received in error,
> > > please destroy and notify the sender. Any use of this email is
> > > prohibited when received in error. Impetus does not represent,
> > > warrant and/or guarantee, that the integrity of this communication
> > > has been maintained nor that the communication is free of errors,
> virus, interception or interference.
> > >
> >
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
RE: Type Casting
Posted by Yash Sharma <ya...@impetus.co.in>.
Great Jinfeng,
The parsing part is something I was stuck yesterday and was trying to figure out. Nice to know you are working on it.
I would then continue with the isCastable check. Would implement it similar to Optiq's Map rather than Hive style.
You can keep working on the parsing part. I would let you know when my part is done.
- Yash
-----Original Message-----
From: Jinfeng Ni [mailto:jinfengni99@gmail.com]
Sent: Monday, November 25, 2013 11:56 PM
To: drill-dev@incubator.apache.org
Subject: Re: Type Casting
Hi Yash,
For the checking of isCastable, I agree that we can not directly use Optiq's SqlAssignmentRules, since the date types are different. But looks like the implementation would be quite similar, as we could use Map<type, Set<type>> to check if one type is allowed to cast to another type.
I'm working on the parser part to support explicit cast function, so that
we can put cast function in logical/physical plan.
How about you continue work on implicit cast and the check of isCastable?
Once both the explicit cast and implicit cast is ready, we can discuss how to merge them together ( I assume implicit will eventually call the explicit cast function to do the real cast).
On Sun, Nov 24, 2013 at 9:27 AM, Jacques Nadeau <ja...@apache.org> wrote:
> For now, you need to do all cast work using logical or physical plans.
> The Optiq currently removes any explicit casts from the logical
> output. Once we depend on an updated Optiq with Mehant's ANY changes,
> we can add explicit cast support back in.
>
> Jinfeng was working on some cast infrastructure as well. Jinfeng, can
> you comment on the second part given the design brainstorming you've done?
>
> thanks,
> Jacques
>
>
> On Sun, Nov 24, 2013 at 1:45 AM, Yash Sharma
> <yash.sharma@impetus.co.in
> >wrote:
>
> > Observations while providing explicit cast functions.
> >
> > 1.
> > On hitting any explicit cast query on Sqlline I get the below results.
> > While debugging the flow I see that there is no lookup happening in
> > FunctionImplementationRegistry for CAST() function, and therefore it
> > is
> not
> > casting the argument and returning the passed argument directly.
> > It seems like some part of query validation is bypassing the lookup
> > for 'cast' function from FunctionRegistry.
> >
> > 0: jdbc:drill:schema=parquet-local> select cast(2.3 as int) from
> > "sample-data/region.parquet";
> > +---------+
> > | EXPR$0 |
> > +---------+
> > | 2.3 |
> > | 2.3 |
> > | 2.3 |
> > | 2.3 |
> > | 2.3 |
> > +---------+
> > 5 rows selected (0.489 seconds)
> >
> >
> >
> > 2.
> > For the 'Is Castable' check, we might have to create our own class
> similar
> > to Optiq's SqlAssignmentRules. Since Drill's datatypes are different
> > from those used in Optiq we might not be able to reuse the class directly.
> > (Ref:
> >
> https://github.com/julianhyde/optiq/blob/master/core/src/main/java/org
> /eigenbase/sql/type/SqlTypeAssignmentRules.java
> > )
> >
> > Alternatively, an approach can be adopted from Hive's
> > org.apache.hadoop.hive.ql.exec.FunctionRegistry's
> > implicitConvertable() method. It uses datatype grouping via enum
> > (PrimitiveGrouping). Then
> > implicitlyConvertable() method checks the common groups to check if
> > types are convertible implicitly.
> >
> > Let me know your thoughts on the two approaches.
> >
> > Regards,
> > Yash
> >
> >
> >
> >
> > ________________________________________
> > From: Julian Hyde [julianhyde@gmail.com]
> > Sent: Friday, November 22, 2013 4:16 AM
> > To: drill-dev@incubator.apache.org
> > Subject: Re: hangout
> >
> > On Nov 20, 2013, at 9:12 PM, Jinfeng Ni <ji...@gmail.com> wrote:
> >
> > > 5. Nullable vs Non-nullable. I think implicit cast probably need
> > > support cast from Non-nullable exp
> > >
> > > Nullable exp. Otherwise, for each operator and each type, we at
> > > least have to implement 4 versions:
> > >
> > > 1) Nullable int + Nullable int
> > >
> > > 2) Nullable int + Non-Nullalbe int
> > >
> > > 3) Non-Nullable int + Nullable int
> > >
> > > 4) Non-Nullable int + Non-Nullalbe int.
> > >
> > > By implicit cast Non-nullable exp into Nullable exp, we only need
> > > define one version: Nullable int + Nullable int.
> > > . How to handle the null input could be specified in
> > > function/operator's implementation.
> >
> > I faced this issue in Optiq's code generation. Since a lot of SQL
> > operators generate null if any of their inputs are null, and since
> > Optiq uses primitive types if it knows an expression cannot be null,
> > I decided
> to
> > implement only one version: Not-Nullable int + Not-Nullable int. If
> either
> > argument is nullable, I add code around it to check for null values
> first.
> >
> > Thus:
> >
> > Integer x;
> > Integer y;
> > if (x == null) {
> > return null;
> > } else {
> > int x_ = x;
> > if (y == null) {
> > return null;
> > } else {
> > int y_ = y;
> > return x_ + y_; // this line is the only one produced by the
> > implementor for '+'
> > }
> > }
> >
> > You can see the gory details in RexToLixTranslator. In the
> > "Expression translate(RexNode expr, RexImpTable.NullAs nullAs)"
> > method, nullAs says whether null should cause the method to return
> > null, true (for
> implementing
> > "x is null"), false, or whether null is simply not possible (because
> we've
> > already dealt with the possibility of nulls.
> >
> > The supporting implementors (that do code generation for each
> > operator or
> > function) are in RexImpTable.
> >
> > Julian
> >
> > ________________________________
> >
> >
> >
> >
> >
> >
> > NOTE: This message may contain information that is confidential,
> > proprietary, privileged or otherwise protected by law. The message
> > is intended solely for the named addressee. If received in error,
> > please destroy and notify the sender. Any use of this email is
> > prohibited when received in error. Impetus does not represent,
> > warrant and/or guarantee, that the integrity of this communication
> > has been maintained nor that the communication is free of errors, virus, interception or interference.
> >
>
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Type Casting
Posted by Jinfeng Ni <ji...@gmail.com>.
Hi Yash,
For the checking of isCastable, I agree that we can not directly use
Optiq's SqlAssignmentRules, since the date types are different. But looks
like the implementation would be quite similar, as we could use Map<type,
Set<type>> to check if one type is allowed to cast to another type.
I'm working on the parser part to support explicit cast function, so that
we can put cast function in logical/physical plan.
How about you continue work on implicit cast and the check of isCastable?
Once both the explicit cast and implicit cast is ready, we can discuss how
to merge them together ( I assume implicit will eventually call the
explicit cast function to do the real cast).
On Sun, Nov 24, 2013 at 9:27 AM, Jacques Nadeau <ja...@apache.org> wrote:
> For now, you need to do all cast work using logical or physical plans. The
> Optiq currently removes any explicit casts from the logical output. Once
> we depend on an updated Optiq with Mehant's ANY changes, we can add
> explicit cast support back in.
>
> Jinfeng was working on some cast infrastructure as well. Jinfeng, can you
> comment on the second part given the design brainstorming you've done?
>
> thanks,
> Jacques
>
>
> On Sun, Nov 24, 2013 at 1:45 AM, Yash Sharma <yash.sharma@impetus.co.in
> >wrote:
>
> > Observations while providing explicit cast functions.
> >
> > 1.
> > On hitting any explicit cast query on Sqlline I get the below results.
> > While debugging the flow I see that there is no lookup happening in
> > FunctionImplementationRegistry for CAST() function, and therefore it is
> not
> > casting the argument and returning the passed argument directly.
> > It seems like some part of query validation is bypassing the lookup for
> > 'cast' function from FunctionRegistry.
> >
> > 0: jdbc:drill:schema=parquet-local> select cast(2.3 as int) from
> > "sample-data/region.parquet";
> > +---------+
> > | EXPR$0 |
> > +---------+
> > | 2.3 |
> > | 2.3 |
> > | 2.3 |
> > | 2.3 |
> > | 2.3 |
> > +---------+
> > 5 rows selected (0.489 seconds)
> >
> >
> >
> > 2.
> > For the 'Is Castable' check, we might have to create our own class
> similar
> > to Optiq's SqlAssignmentRules. Since Drill's datatypes are different from
> > those used in Optiq we might not be able to reuse the class directly.
> > (Ref:
> >
> https://github.com/julianhyde/optiq/blob/master/core/src/main/java/org/eigenbase/sql/type/SqlTypeAssignmentRules.java
> > )
> >
> > Alternatively, an approach can be adopted from Hive's
> > org.apache.hadoop.hive.ql.exec.FunctionRegistry's implicitConvertable()
> > method. It uses datatype grouping via enum (PrimitiveGrouping). Then
> > implicitlyConvertable() method checks the common groups to check if types
> > are convertible implicitly.
> >
> > Let me know your thoughts on the two approaches.
> >
> > Regards,
> > Yash
> >
> >
> >
> >
> > ________________________________________
> > From: Julian Hyde [julianhyde@gmail.com]
> > Sent: Friday, November 22, 2013 4:16 AM
> > To: drill-dev@incubator.apache.org
> > Subject: Re: hangout
> >
> > On Nov 20, 2013, at 9:12 PM, Jinfeng Ni <ji...@gmail.com> wrote:
> >
> > > 5. Nullable vs Non-nullable. I think implicit cast probably need
> > > support cast from Non-nullable exp
> > >
> > > Nullable exp. Otherwise, for each operator and each type, we at least
> > > have to implement 4 versions:
> > >
> > > 1) Nullable int + Nullable int
> > >
> > > 2) Nullable int + Non-Nullalbe int
> > >
> > > 3) Non-Nullable int + Nullable int
> > >
> > > 4) Non-Nullable int + Non-Nullalbe int.
> > >
> > > By implicit cast Non-nullable exp into Nullable exp, we only need
> > > define one version: Nullable int + Nullable int.
> > > . How to handle the null input could be specified in
> > > function/operator's implementation.
> >
> > I faced this issue in Optiq's code generation. Since a lot of SQL
> > operators generate null if any of their inputs are null, and since Optiq
> > uses primitive types if it knows an expression cannot be null, I decided
> to
> > implement only one version: Not-Nullable int + Not-Nullable int. If
> either
> > argument is nullable, I add code around it to check for null values
> first.
> >
> > Thus:
> >
> > Integer x;
> > Integer y;
> > if (x == null) {
> > return null;
> > } else {
> > int x_ = x;
> > if (y == null) {
> > return null;
> > } else {
> > int y_ = y;
> > return x_ + y_; // this line is the only one produced by the
> > implementor for '+'
> > }
> > }
> >
> > You can see the gory details in RexToLixTranslator. In the "Expression
> > translate(RexNode expr, RexImpTable.NullAs nullAs)" method, nullAs says
> > whether null should cause the method to return null, true (for
> implementing
> > "x is null"), false, or whether null is simply not possible (because
> we've
> > already dealt with the possibility of nulls.
> >
> > The supporting implementors (that do code generation for each operator or
> > function) are in RexImpTable.
> >
> > Julian
> >
> > ________________________________
> >
> >
> >
> >
> >
> >
> > NOTE: This message may contain information that is confidential,
> > proprietary, privileged or otherwise protected by law. The message is
> > intended solely for the named addressee. If received in error, please
> > destroy and notify the sender. Any use of this email is prohibited when
> > received in error. Impetus does not represent, warrant and/or guarantee,
> > that the integrity of this communication has been maintained nor that the
> > communication is free of errors, virus, interception or interference.
> >
>
Re: Type Casting
Posted by Jacques Nadeau <ja...@apache.org>.
For now, you need to do all cast work using logical or physical plans. The
Optiq currently removes any explicit casts from the logical output. Once
we depend on an updated Optiq with Mehant's ANY changes, we can add
explicit cast support back in.
Jinfeng was working on some cast infrastructure as well. Jinfeng, can you
comment on the second part given the design brainstorming you've done?
thanks,
Jacques
On Sun, Nov 24, 2013 at 1:45 AM, Yash Sharma <ya...@impetus.co.in>wrote:
> Observations while providing explicit cast functions.
>
> 1.
> On hitting any explicit cast query on Sqlline I get the below results.
> While debugging the flow I see that there is no lookup happening in
> FunctionImplementationRegistry for CAST() function, and therefore it is not
> casting the argument and returning the passed argument directly.
> It seems like some part of query validation is bypassing the lookup for
> 'cast' function from FunctionRegistry.
>
> 0: jdbc:drill:schema=parquet-local> select cast(2.3 as int) from
> "sample-data/region.parquet";
> +---------+
> | EXPR$0 |
> +---------+
> | 2.3 |
> | 2.3 |
> | 2.3 |
> | 2.3 |
> | 2.3 |
> +---------+
> 5 rows selected (0.489 seconds)
>
>
>
> 2.
> For the 'Is Castable' check, we might have to create our own class similar
> to Optiq's SqlAssignmentRules. Since Drill's datatypes are different from
> those used in Optiq we might not be able to reuse the class directly.
> (Ref:
> https://github.com/julianhyde/optiq/blob/master/core/src/main/java/org/eigenbase/sql/type/SqlTypeAssignmentRules.java
> )
>
> Alternatively, an approach can be adopted from Hive's
> org.apache.hadoop.hive.ql.exec.FunctionRegistry's implicitConvertable()
> method. It uses datatype grouping via enum (PrimitiveGrouping). Then
> implicitlyConvertable() method checks the common groups to check if types
> are convertible implicitly.
>
> Let me know your thoughts on the two approaches.
>
> Regards,
> Yash
>
>
>
>
> ________________________________________
> From: Julian Hyde [julianhyde@gmail.com]
> Sent: Friday, November 22, 2013 4:16 AM
> To: drill-dev@incubator.apache.org
> Subject: Re: hangout
>
> On Nov 20, 2013, at 9:12 PM, Jinfeng Ni <ji...@gmail.com> wrote:
>
> > 5. Nullable vs Non-nullable. I think implicit cast probably need
> > support cast from Non-nullable exp
> >
> > Nullable exp. Otherwise, for each operator and each type, we at least
> > have to implement 4 versions:
> >
> > 1) Nullable int + Nullable int
> >
> > 2) Nullable int + Non-Nullalbe int
> >
> > 3) Non-Nullable int + Nullable int
> >
> > 4) Non-Nullable int + Non-Nullalbe int.
> >
> > By implicit cast Non-nullable exp into Nullable exp, we only need
> > define one version: Nullable int + Nullable int.
> > . How to handle the null input could be specified in
> > function/operator's implementation.
>
> I faced this issue in Optiq's code generation. Since a lot of SQL
> operators generate null if any of their inputs are null, and since Optiq
> uses primitive types if it knows an expression cannot be null, I decided to
> implement only one version: Not-Nullable int + Not-Nullable int. If either
> argument is nullable, I add code around it to check for null values first.
>
> Thus:
>
> Integer x;
> Integer y;
> if (x == null) {
> return null;
> } else {
> int x_ = x;
> if (y == null) {
> return null;
> } else {
> int y_ = y;
> return x_ + y_; // this line is the only one produced by the
> implementor for '+'
> }
> }
>
> You can see the gory details in RexToLixTranslator. In the "Expression
> translate(RexNode expr, RexImpTable.NullAs nullAs)" method, nullAs says
> whether null should cause the method to return null, true (for implementing
> "x is null"), false, or whether null is simply not possible (because we've
> already dealt with the possibility of nulls.
>
> The supporting implementors (that do code generation for each operator or
> function) are in RexImpTable.
>
> Julian
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>