You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lens.apache.org by Sushil Mohanty <su...@gmail.com> on 2015/08/06 06:54:16 UTC
Review Request 37163: LENS-714 : Replace dimension tables with
subquery in Columnar Rewriter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37163/
-----------------------------------------------------------
Review request for lens and Amareshwari Sriramadasu.
Bugs: LENS-714
https://issues.apache.org/jira/browse/LENS-714
Repository: lens
Description
-------
To improve performance of SQL queries in columnar db chnaging the tables with subqueries with only required fields for a query.
Diffs
-----
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java 0556d0c
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 1b61010
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 1136e7c
Diff: https://reviews.apache.org/r/37163/diff/
Testing
-------
Modified unit tests accordingly. All tests passes.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [4.901s]
[INFO] Lens .............................................. SUCCESS [0.278s]
[INFO] Lens API .......................................... SUCCESS [23.143s]
[INFO] Lens API for server and extensions ................ SUCCESS [19.000s]
[INFO] Lens Cube ......................................... SUCCESS [8:35.786s]
[INFO] Lens DB storage ................................... SUCCESS [31.557s]
[INFO] Lens Query Library ................................ SUCCESS [9.908s]
[INFO] Lens Hive Driver .................................. SUCCESS [3:57.883s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [42.114s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [14.187s]
[INFO] Lens Server ....................................... SUCCESS [13:54.061s]
[INFO] Lens client ....................................... SUCCESS [42.943s]
[INFO] Lens CLI .......................................... SUCCESS [3:49.813s]
[INFO] Lens Examples ..................................... SUCCESS [2.457s]
[INFO] Lens Distribution ................................. SUCCESS [2.274s]
[INFO] Lens ML Lib ....................................... SUCCESS [6:13.820s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [11.709s]
[INFO] Lens Regression ................................... SUCCESS [3.722s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 40:01.357s
[INFO] Finished at: Thu Aug 06 09:29:43 IST 2015
[INFO] Final Memory: 167M/1839M
[INFO] ------------------------------------------------------------------------
Thanks,
Sushil Mohanty
Re: Review Request 37163: LENS-714 : Replace dimension tables with
subquery in Columnar Rewriter
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37163/#review94526
-----------------------------------------------------------
Ship it!
Ship It!
- Amareshwari Sriramadasu
On Aug. 7, 2015, 10:33 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37163/
> -----------------------------------------------------------
>
> (Updated Aug. 7, 2015, 10:33 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-714
> https://issues.apache.org/jira/browse/LENS-714
>
>
> Repository: lens
>
>
> Description
> -------
>
> To improve performance of SQL queries in columnar db chnaging the tables with subqueries with only required fields for a query.
>
>
> Diffs
> -----
>
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java 0556d0c
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 1b61010
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 1136e7c
>
> Diff: https://reviews.apache.org/r/37163/diff/
>
>
> Testing
> -------
>
> Modified unit tests accordingly. All tests passes.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [4.901s]
> [INFO] Lens .............................................. SUCCESS [0.278s]
> [INFO] Lens API .......................................... SUCCESS [23.143s]
> [INFO] Lens API for server and extensions ................ SUCCESS [19.000s]
> [INFO] Lens Cube ......................................... SUCCESS [8:35.786s]
> [INFO] Lens DB storage ................................... SUCCESS [31.557s]
> [INFO] Lens Query Library ................................ SUCCESS [9.908s]
> [INFO] Lens Hive Driver .................................. SUCCESS [3:57.883s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [42.114s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [14.187s]
> [INFO] Lens Server ....................................... SUCCESS [13:54.061s]
> [INFO] Lens client ....................................... SUCCESS [42.943s]
> [INFO] Lens CLI .......................................... SUCCESS [3:49.813s]
> [INFO] Lens Examples ..................................... SUCCESS [2.457s]
> [INFO] Lens Distribution ................................. SUCCESS [2.274s]
> [INFO] Lens ML Lib ....................................... SUCCESS [6:13.820s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.709s]
> [INFO] Lens Regression ................................... SUCCESS [3.722s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 40:01.357s
> [INFO] Finished at: Thu Aug 06 09:29:43 IST 2015
> [INFO] Final Memory: 167M/1839M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 37163: LENS-714 : Replace dimension tables with
subquery in Columnar Rewriter
Posted by Sushil Mohanty <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37163/
-----------------------------------------------------------
(Updated Aug. 7, 2015, 12:24 p.m.)
Review request for lens and Amareshwari Sriramadasu.
Changes
-------
Addressed review comment. Tests are running fine.
Bugs: LENS-714
https://issues.apache.org/jira/browse/LENS-714
Repository: lens
Description
-------
To improve performance of SQL queries in columnar db chnaging the tables with subqueries with only required fields for a query.
Diffs (updated)
-----
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java 0556d0c
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 1b61010
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 1136e7c
Diff: https://reviews.apache.org/r/37163/diff/
Testing
-------
Modified unit tests accordingly. All tests passes.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [4.901s]
[INFO] Lens .............................................. SUCCESS [0.278s]
[INFO] Lens API .......................................... SUCCESS [23.143s]
[INFO] Lens API for server and extensions ................ SUCCESS [19.000s]
[INFO] Lens Cube ......................................... SUCCESS [8:35.786s]
[INFO] Lens DB storage ................................... SUCCESS [31.557s]
[INFO] Lens Query Library ................................ SUCCESS [9.908s]
[INFO] Lens Hive Driver .................................. SUCCESS [3:57.883s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [42.114s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [14.187s]
[INFO] Lens Server ....................................... SUCCESS [13:54.061s]
[INFO] Lens client ....................................... SUCCESS [42.943s]
[INFO] Lens CLI .......................................... SUCCESS [3:49.813s]
[INFO] Lens Examples ..................................... SUCCESS [2.457s]
[INFO] Lens Distribution ................................. SUCCESS [2.274s]
[INFO] Lens ML Lib ....................................... SUCCESS [6:13.820s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [11.709s]
[INFO] Lens Regression ................................... SUCCESS [3.722s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 40:01.357s
[INFO] Finished at: Thu Aug 06 09:29:43 IST 2015
[INFO] Final Memory: 167M/1839M
[INFO] ------------------------------------------------------------------------
Thanks,
Sushil Mohanty
Re: Review Request 37163: LENS-714 : Replace dimension tables with
subquery in Columnar Rewriter
Posted by Rajat Khandelwal <ra...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37163/#review94525
-----------------------------------------------------------
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java (line 307)
<https://reviews.apache.org/r/37163/#comment149148>
we can probably split on `" +"`
- Rajat Khandelwal
On Aug. 7, 2015, 4:03 p.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37163/
> -----------------------------------------------------------
>
> (Updated Aug. 7, 2015, 4:03 p.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-714
> https://issues.apache.org/jira/browse/LENS-714
>
>
> Repository: lens
>
>
> Description
> -------
>
> To improve performance of SQL queries in columnar db chnaging the tables with subqueries with only required fields for a query.
>
>
> Diffs
> -----
>
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java 0556d0c
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 1b61010
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 1136e7c
>
> Diff: https://reviews.apache.org/r/37163/diff/
>
>
> Testing
> -------
>
> Modified unit tests accordingly. All tests passes.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [4.901s]
> [INFO] Lens .............................................. SUCCESS [0.278s]
> [INFO] Lens API .......................................... SUCCESS [23.143s]
> [INFO] Lens API for server and extensions ................ SUCCESS [19.000s]
> [INFO] Lens Cube ......................................... SUCCESS [8:35.786s]
> [INFO] Lens DB storage ................................... SUCCESS [31.557s]
> [INFO] Lens Query Library ................................ SUCCESS [9.908s]
> [INFO] Lens Hive Driver .................................. SUCCESS [3:57.883s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [42.114s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [14.187s]
> [INFO] Lens Server ....................................... SUCCESS [13:54.061s]
> [INFO] Lens client ....................................... SUCCESS [42.943s]
> [INFO] Lens CLI .......................................... SUCCESS [3:49.813s]
> [INFO] Lens Examples ..................................... SUCCESS [2.457s]
> [INFO] Lens Distribution ................................. SUCCESS [2.274s]
> [INFO] Lens ML Lib ....................................... SUCCESS [6:13.820s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.709s]
> [INFO] Lens Regression ................................... SUCCESS [3.722s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 40:01.357s
> [INFO] Finished at: Thu Aug 06 09:29:43 IST 2015
> [INFO] Final Memory: 167M/1839M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>
Re: Review Request 37163: LENS-714 : Replace dimension tables with
subquery in Columnar Rewriter
Posted by Sushil Mohanty <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37163/
-----------------------------------------------------------
(Updated Aug. 7, 2015, 10:33 a.m.)
Review request for lens and Amareshwari Sriramadasu.
Changes
-------
Addressed review comments. All jdbc unit tests are running fine. Will update the full test result shortly.
Bugs: LENS-714
https://issues.apache.org/jira/browse/LENS-714
Repository: lens
Description
-------
To improve performance of SQL queries in columnar db chnaging the tables with subqueries with only required fields for a query.
Diffs (updated)
-----
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java 0556d0c
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 1b61010
lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 1136e7c
Diff: https://reviews.apache.org/r/37163/diff/
Testing
-------
Modified unit tests accordingly. All tests passes.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Lens Checkstyle Rules ............................. SUCCESS [4.901s]
[INFO] Lens .............................................. SUCCESS [0.278s]
[INFO] Lens API .......................................... SUCCESS [23.143s]
[INFO] Lens API for server and extensions ................ SUCCESS [19.000s]
[INFO] Lens Cube ......................................... SUCCESS [8:35.786s]
[INFO] Lens DB storage ................................... SUCCESS [31.557s]
[INFO] Lens Query Library ................................ SUCCESS [9.908s]
[INFO] Lens Hive Driver .................................. SUCCESS [3:57.883s]
[INFO] Lens Driver for JDBC .............................. SUCCESS [42.114s]
[INFO] Lens Elastic Search Driver ........................ SUCCESS [14.187s]
[INFO] Lens Server ....................................... SUCCESS [13:54.061s]
[INFO] Lens client ....................................... SUCCESS [42.943s]
[INFO] Lens CLI .......................................... SUCCESS [3:49.813s]
[INFO] Lens Examples ..................................... SUCCESS [2.457s]
[INFO] Lens Distribution ................................. SUCCESS [2.274s]
[INFO] Lens ML Lib ....................................... SUCCESS [6:13.820s]
[INFO] Lens ML Ext Distribution .......................... SUCCESS [11.709s]
[INFO] Lens Regression ................................... SUCCESS [3.722s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 40:01.357s
[INFO] Finished at: Thu Aug 06 09:29:43 IST 2015
[INFO] Final Memory: 167M/1839M
[INFO] ------------------------------------------------------------------------
Thanks,
Sushil Mohanty
Re: Review Request 37163: LENS-714 : Replace dimension tables with
subquery in Columnar Rewriter
Posted by Amareshwari Sriramadasu <am...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37163/#review94492
-----------------------------------------------------------
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java (line 274)
<https://reviews.apache.org/r/37163/#comment149119>
Is this required? I dont see how this is handled in rest of the code
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java (line 308)
<https://reviews.apache.org/r/37163/#comment149120>
What is this split on space doing here? Can you add comments to understand it better.
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java (line 538)
<https://reviews.apache.org/r/37163/#comment149118>
Can you remove printAST here?
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java (line 539)
<https://reviews.apache.org/r/37163/#comment149121>
This is assuming column is alwasys specified by table.column in the query. Can you add it as comment, if assumption cannot be removed?
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java (line 547)
<https://reviews.apache.org/r/37163/#comment149122>
!column.equals(null) can be put as column != null
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java (line 548)
<https://reviews.apache.org/r/37163/#comment149123>
Can the creation of new HashSet be moved inside the 'if' block below?
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java (line 556)
<https://reviews.apache.org/r/37163/#comment149124>
This is redundant. Can be removed.
lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java (line 577)
<https://reviews.apache.org/r/37163/#comment149126>
Can we replace toString().replace().replace() with StringUtils.join(cols, ",") ?
- Amareshwari Sriramadasu
On Aug. 6, 2015, 4:54 a.m., Sushil Mohanty wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37163/
> -----------------------------------------------------------
>
> (Updated Aug. 6, 2015, 4:54 a.m.)
>
>
> Review request for lens and Amareshwari Sriramadasu.
>
>
> Bugs: LENS-714
> https://issues.apache.org/jira/browse/LENS-714
>
>
> Repository: lens
>
>
> Description
> -------
>
> To improve performance of SQL queries in columnar db chnaging the tables with subqueries with only required fields for a query.
>
>
> Diffs
> -----
>
> lens-driver-jdbc/src/main/java/org/apache/lens/driver/jdbc/ColumnarSQLRewriter.java 0556d0c
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestColumnarSQLRewriter.java 1b61010
> lens-driver-jdbc/src/test/java/org/apache/lens/driver/jdbc/TestJDBCFinal.java 1136e7c
>
> Diff: https://reviews.apache.org/r/37163/diff/
>
>
> Testing
> -------
>
> Modified unit tests accordingly. All tests passes.
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Lens Checkstyle Rules ............................. SUCCESS [4.901s]
> [INFO] Lens .............................................. SUCCESS [0.278s]
> [INFO] Lens API .......................................... SUCCESS [23.143s]
> [INFO] Lens API for server and extensions ................ SUCCESS [19.000s]
> [INFO] Lens Cube ......................................... SUCCESS [8:35.786s]
> [INFO] Lens DB storage ................................... SUCCESS [31.557s]
> [INFO] Lens Query Library ................................ SUCCESS [9.908s]
> [INFO] Lens Hive Driver .................................. SUCCESS [3:57.883s]
> [INFO] Lens Driver for JDBC .............................. SUCCESS [42.114s]
> [INFO] Lens Elastic Search Driver ........................ SUCCESS [14.187s]
> [INFO] Lens Server ....................................... SUCCESS [13:54.061s]
> [INFO] Lens client ....................................... SUCCESS [42.943s]
> [INFO] Lens CLI .......................................... SUCCESS [3:49.813s]
> [INFO] Lens Examples ..................................... SUCCESS [2.457s]
> [INFO] Lens Distribution ................................. SUCCESS [2.274s]
> [INFO] Lens ML Lib ....................................... SUCCESS [6:13.820s]
> [INFO] Lens ML Ext Distribution .......................... SUCCESS [11.709s]
> [INFO] Lens Regression ................................... SUCCESS [3.722s]
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 40:01.357s
> [INFO] Finished at: Thu Aug 06 09:29:43 IST 2015
> [INFO] Final Memory: 167M/1839M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Sushil Mohanty
>
>