You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Sean Hsuan-Yi Chu <hs...@usc.edu> on 2015/01/02 23:09:08 UTC
Review Request 29549: DRILL-1874: Text reader should allow treating
empty fields as NULLs instead of empty string
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29549/
-----------------------------------------------------------
Review request for drill, Aman Sinha and Jinfeng Ni.
Bugs: DRILL-1874
https://issues.apache.org/jira/browse/DRILL-1874
Repository: drill-git
Description
-------
Users can choose to use safe (for varchar inputs)/default cast functions by setting sesseion variable ("alter session set `planner.safe_cast_varchar` = true;").
Add comments, Clean Codes
Diffs
-----
common/src/main/java/org/apache/drill/common/expression/fn/CastFunctions.java 61d149d
exec/java-exec/src/main/codegen/data/Casts.tdd 36c90c9
exec/java-exec/src/main/codegen/templates/CastSafeFunctions.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java b390cd5
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 0f40958
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java fa983aa
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 34729e2
exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java e0cce8b
Diff: https://reviews.apache.org/r/29549/diff/
Testing
-------
unit tests,
functional tests,
tpch tests
Thanks,
Sean Hsuan-Yi Chu
Re: Review Request 29549: DRILL-1874: Text reader should allow
treating empty fields as NULLs instead of empty string
Posted by Sean Hsuan-Yi Chu <hs...@usc.edu>.
> On Jan. 2, 2015, 11:58 p.m., Jacques Nadeau wrote:
> > I'm also confused. I was expecting cast safe functions to convert an empty string into a null numeric value. Is that functionality here?
Indeed, it is. But I also allowed users to choose to use the original cast function:
More precisely, if you type in "alter session set planner.safe_cast_varchar = true", the new safe cast funtion (which casts an empty string into a null numeric value).
Otherwise, planner.safe_cast_varchar is false by default. So the orignal cast function is chosen.
- Sean Hsuan-Yi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29549/#review66566
-----------------------------------------------------------
On Jan. 2, 2015, 10:09 p.m., Sean Hsuan-Yi Chu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29549/
> -----------------------------------------------------------
>
> (Updated Jan. 2, 2015, 10:09 p.m.)
>
>
> Review request for drill, Aman Sinha and Jinfeng Ni.
>
>
> Bugs: DRILL-1874
> https://issues.apache.org/jira/browse/DRILL-1874
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Users can choose to use safe (for varchar inputs)/default cast functions by setting sesseion variable ("alter session set `planner.safe_cast_varchar` = true;").
>
> Add comments, Clean Codes
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/drill/common/expression/fn/CastFunctions.java 61d149d
> exec/java-exec/src/main/codegen/data/Casts.tdd 36c90c9
> exec/java-exec/src/main/codegen/templates/CastSafeFunctions.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java b390cd5
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 0f40958
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java fa983aa
> exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 34729e2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java e0cce8b
>
> Diff: https://reviews.apache.org/r/29549/diff/
>
>
> Testing
> -------
>
> unit tests,
> functional tests,
> tpch tests
>
>
> Thanks,
>
> Sean Hsuan-Yi Chu
>
>
Re: Review Request 29549: DRILL-1874: Text reader should allow
treating empty fields as NULLs instead of empty string
Posted by Jacques Nadeau <ja...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29549/#review66566
-----------------------------------------------------------
I'm also confused. I was expecting cast safe functions to convert an empty string into a null numeric value. Is that functionality here?
- Jacques Nadeau
On Jan. 2, 2015, 10:09 p.m., Sean Hsuan-Yi Chu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29549/
> -----------------------------------------------------------
>
> (Updated Jan. 2, 2015, 10:09 p.m.)
>
>
> Review request for drill, Aman Sinha and Jinfeng Ni.
>
>
> Bugs: DRILL-1874
> https://issues.apache.org/jira/browse/DRILL-1874
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Users can choose to use safe (for varchar inputs)/default cast functions by setting sesseion variable ("alter session set `planner.safe_cast_varchar` = true;").
>
> Add comments, Clean Codes
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/drill/common/expression/fn/CastFunctions.java 61d149d
> exec/java-exec/src/main/codegen/data/Casts.tdd 36c90c9
> exec/java-exec/src/main/codegen/templates/CastSafeFunctions.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java b390cd5
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 0f40958
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java fa983aa
> exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 34729e2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java e0cce8b
>
> Diff: https://reviews.apache.org/r/29549/diff/
>
>
> Testing
> -------
>
> unit tests,
> functional tests,
> tpch tests
>
>
> Thanks,
>
> Sean Hsuan-Yi Chu
>
>
Re: Review Request 29549: DRILL-1874: Text reader should allow
treating empty fields as NULLs instead of empty string
Posted by Sean Hsuan-Yi Chu <hs...@usc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29549/
-----------------------------------------------------------
(Updated Jan. 5, 2015, 4:55 a.m.)
Review request for drill, Aman Sinha and Jinfeng Ni.
Changes
-------
When dealing with nullable type, this version first checked if the input is set (isSet is 0 or not).
Bugs: DRILL-1874
https://issues.apache.org/jira/browse/DRILL-1874
Repository: drill-git
Description
-------
Users can choose to use safe (for varchar inputs)/default cast functions by setting sesseion variable ("alter session set `planner.safe_cast_varchar` = true;").
Add comments, Clean Codes
Diffs (updated)
-----
common/src/main/java/org/apache/drill/common/expression/fn/CastFunctions.java 61d149d
exec/java-exec/src/main/codegen/data/Casts.tdd 36c90c9
exec/java-exec/src/main/codegen/templates/CastSafeFromNonnullableFunctions.java PRE-CREATION
exec/java-exec/src/main/codegen/templates/CastSafeFromNullableFunctions.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java b390cd5
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 0f40958
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java fa983aa
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 34729e2
exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java e0cce8b
Diff: https://reviews.apache.org/r/29549/diff/
Testing
-------
unit tests,
functional tests,
tpch tests
Thanks,
Sean Hsuan-Yi Chu
Re: Review Request 29549: DRILL-1874: Text reader should allow
treating empty fields as NULLs instead of empty string
Posted by Sean Hsuan-Yi Chu <hs...@usc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29549/
-----------------------------------------------------------
(Updated Jan. 4, 2015, 11:46 p.m.)
Review request for drill, Aman Sinha and Jinfeng Ni.
Bugs: DRILL-1874
https://issues.apache.org/jira/browse/DRILL-1874
Repository: drill-git
Description
-------
Users can choose to use safe (for varchar inputs)/default cast functions by setting sesseion variable ("alter session set `planner.safe_cast_varchar` = true;").
Add comments, Clean Codes
Diffs (updated)
-----
common/src/main/java/org/apache/drill/common/expression/fn/CastFunctions.java 61d149d
exec/java-exec/src/main/codegen/data/Casts.tdd 36c90c9
exec/java-exec/src/main/codegen/templates/CastSafeFromNonnullableFunctions.java PRE-CREATION
exec/java-exec/src/main/codegen/templates/CastSafeFromNullableFunctions.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java b390cd5
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 0f40958
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java fa983aa
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 34729e2
exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java e0cce8b
Diff: https://reviews.apache.org/r/29549/diff/
Testing
-------
unit tests,
functional tests,
tpch tests
Thanks,
Sean Hsuan-Yi Chu
Re: Review Request 29549: DRILL-1874: Text reader should allow
treating empty fields as NULLs instead of empty string
Posted by Sean Hsuan-Yi Chu <hs...@usc.edu>.
> On Jan. 2, 2015, 11:32 p.m., Jinfeng Ni wrote:
> > exec/java-exec/src/main/codegen/templates/CastSafeFunctions.java, line 50
> > <https://reviews.apache.org/r/29549/diff/1/?file=805817#file805817line50>
> >
> > Will this new castSafe function work,if the input is not-nullable varchar (since you are using NullHandling.INTERNAL) ?
I add another template () to deal with non-nullable varchar input
=> CastSafeFromNullableFunctions.java is for Nullable
=> CastSafeFromNonnullableFunctions.java is for non-nullable
- Sean Hsuan-Yi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29549/#review66562
-----------------------------------------------------------
On Jan. 4, 2015, 11:46 p.m., Sean Hsuan-Yi Chu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29549/
> -----------------------------------------------------------
>
> (Updated Jan. 4, 2015, 11:46 p.m.)
>
>
> Review request for drill, Aman Sinha and Jinfeng Ni.
>
>
> Bugs: DRILL-1874
> https://issues.apache.org/jira/browse/DRILL-1874
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Users can choose to use safe (for varchar inputs)/default cast functions by setting sesseion variable ("alter session set `planner.safe_cast_varchar` = true;").
>
> Add comments, Clean Codes
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/drill/common/expression/fn/CastFunctions.java 61d149d
> exec/java-exec/src/main/codegen/data/Casts.tdd 36c90c9
> exec/java-exec/src/main/codegen/templates/CastSafeFromNonnullableFunctions.java PRE-CREATION
> exec/java-exec/src/main/codegen/templates/CastSafeFromNullableFunctions.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java b390cd5
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 0f40958
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java fa983aa
> exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 34729e2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java e0cce8b
>
> Diff: https://reviews.apache.org/r/29549/diff/
>
>
> Testing
> -------
>
> unit tests,
> functional tests,
> tpch tests
>
>
> Thanks,
>
> Sean Hsuan-Yi Chu
>
>
Re: Review Request 29549: DRILL-1874: Text reader should allow
treating empty fields as NULLs instead of empty string
Posted by Jinfeng Ni <jn...@maprtech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29549/#review66562
-----------------------------------------------------------
exec/java-exec/src/main/codegen/templates/CastSafeFunctions.java
<https://reviews.apache.org/r/29549/#comment110134>
Will this new castSafe function work,if the input is not-nullable varchar (since you are using NullHandling.INTERNAL) ?
- Jinfeng Ni
On Jan. 2, 2015, 2:09 p.m., Sean Hsuan-Yi Chu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29549/
> -----------------------------------------------------------
>
> (Updated Jan. 2, 2015, 2:09 p.m.)
>
>
> Review request for drill, Aman Sinha and Jinfeng Ni.
>
>
> Bugs: DRILL-1874
> https://issues.apache.org/jira/browse/DRILL-1874
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Users can choose to use safe (for varchar inputs)/default cast functions by setting sesseion variable ("alter session set `planner.safe_cast_varchar` = true;").
>
> Add comments, Clean Codes
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/drill/common/expression/fn/CastFunctions.java 61d149d
> exec/java-exec/src/main/codegen/data/Casts.tdd 36c90c9
> exec/java-exec/src/main/codegen/templates/CastSafeFunctions.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java b390cd5
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 0f40958
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java fa983aa
> exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 34729e2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java e0cce8b
>
> Diff: https://reviews.apache.org/r/29549/diff/
>
>
> Testing
> -------
>
> unit tests,
> functional tests,
> tpch tests
>
>
> Thanks,
>
> Sean Hsuan-Yi Chu
>
>
Re: Review Request 29549: DRILL-1874: Text reader should allow
treating empty fields as NULLs instead of empty string
Posted by Jacques Nadeau <ja...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29549/#review66561
-----------------------------------------------------------
Since these are function replacements, have you considered using session option detection in the FunctionImplementationRegistry? It doesn't seem like we should be custom handling these in materialization.
- Jacques Nadeau
On Jan. 2, 2015, 10:09 p.m., Sean Hsuan-Yi Chu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29549/
> -----------------------------------------------------------
>
> (Updated Jan. 2, 2015, 10:09 p.m.)
>
>
> Review request for drill, Aman Sinha and Jinfeng Ni.
>
>
> Bugs: DRILL-1874
> https://issues.apache.org/jira/browse/DRILL-1874
>
>
> Repository: drill-git
>
>
> Description
> -------
>
> Users can choose to use safe (for varchar inputs)/default cast functions by setting sesseion variable ("alter session set `planner.safe_cast_varchar` = true;").
>
> Add comments, Clean Codes
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/drill/common/expression/fn/CastFunctions.java 61d149d
> exec/java-exec/src/main/codegen/data/Casts.tdd 36c90c9
> exec/java-exec/src/main/codegen/templates/CastSafeFunctions.java PRE-CREATION
> exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java b390cd5
> exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 0f40958
> exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java fa983aa
> exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 34729e2
> exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java e0cce8b
>
> Diff: https://reviews.apache.org/r/29549/diff/
>
>
> Testing
> -------
>
> unit tests,
> functional tests,
> tpch tests
>
>
> Thanks,
>
> Sean Hsuan-Yi Chu
>
>
Re: Review Request 29549: DRILL-1874: Text reader should allow
treating empty fields as NULLs instead of empty string
Posted by Sean Hsuan-Yi Chu <hs...@usc.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29549/
-----------------------------------------------------------
(Updated Jan. 2, 2015, 10:09 p.m.)
Review request for drill, Aman Sinha and Jinfeng Ni.
Bugs: DRILL-1874
https://issues.apache.org/jira/browse/DRILL-1874
Repository: drill-git
Description
-------
Users can choose to use safe (for varchar inputs)/default cast functions by setting sesseion variable ("alter session set `planner.safe_cast_varchar` = true;").
Add comments, Clean Codes
Diffs
-----
common/src/main/java/org/apache/drill/common/expression/fn/CastFunctions.java 61d149d
exec/java-exec/src/main/codegen/data/Casts.tdd 36c90c9
exec/java-exec/src/main/codegen/templates/CastSafeFunctions.java PRE-CREATION
exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java b390cd5
exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 0f40958
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/ProjectRecordBatch.java fa983aa
exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 34729e2
exec/java-exec/src/main/java/org/apache/drill/exec/store/text/DrillTextRecordReader.java e0cce8b
Diff: https://reviews.apache.org/r/29549/diff/
Testing
-------
unit tests,
functional tests,
tpch tests
Thanks,
Sean Hsuan-Yi Chu