You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2015/11/30 03:57:11 UTC
[jira] [Commented] (SPARK-9182) filter and groupBy on DataFrames
are not passed through to jdbc source
[ https://issues.apache.org/jira/browse/SPARK-9182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15031266#comment-15031266 ]
Hyukjin Kwon commented on SPARK-9182:
-------------------------------------
Just a thought, Currently {{DataSourceStrategy.translateFilter}} just converts {{expressions.Filter}} to {{sources.Filter}} which are commonly convertable.
However, for other datasources, more or less filters could be processed. As far as I know, to exclude some filters, {{unhanldedFilters}} interface is added by https://issues.apache.org/jira/browse/SPARK-10978.
To include some more filters for example for JDBC and others such as Elasticsearch or Solr, should we better just convert all {{expressions.Filter}} to {{sources.Filter}} to hide the internals and then let the {{unhanldedFilters}} select the filters that it can process?
Even though adding this logic to the Spark internal datasources (namely correcting the Parquet or ORC datasources to get rid of duplicated filters) should also be done, this still would be advantageous as this would remove Spark-side filtering (currently, the internal datasources filter data twice at Spark side and also datasource side).
> filter and groupBy on DataFrames are not passed through to jdbc source
> ----------------------------------------------------------------------
>
> Key: SPARK-9182
> URL: https://issues.apache.org/jira/browse/SPARK-9182
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.1
> Reporter: Greg Rahn
> Assignee: Yijie Shen
> Priority: Critical
>
> When running all of these API calls, the only one that passes the filter through to the backend jdbc source is equality. All filters in these commands should be able to be passed through to the jdbc database source.
> {code}
> val url="jdbc:postgresql:grahn"
> val prop = new java.util.Properties
> val emp = sqlContext.read.jdbc(url, "emp", prop)
> emp.filter(emp("sal") === 5000).show()
> emp.filter(emp("sal") < 5000).show()
> emp.filter("sal = 3000").show()
> emp.filter("sal > 2500").show()
> emp.filter("sal >= 2500").show()
> emp.filter("sal < 2500").show()
> emp.filter("sal <= 2500").show()
> emp.filter("sal != 3000").show()
> emp.filter("sal between 3000 and 5000").show()
> emp.filter("ename in ('SCOTT','BLAKE')").show()
> {code}
> We see from the PostgreSQL query log the following is run, and see that only equality predicates are passed through.
> {code}
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp WHERE sal = 5000
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp WHERE sal = 3000
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> LOG: execute <unnamed>: SET extra_float_digits = 3
> LOG: execute <unnamed>: SELECT "empno","ename","job","mgr","hiredate","sal","comm","deptno" FROM emp
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org