You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Lai Zhou (JIRA)" <ji...@apache.org> on 2019/06/11 11:52:00 UTC
[jira] [Comment Edited] (SPARK-9983) Local physical operators for
query execution
[ https://issues.apache.org/jira/browse/SPARK-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16860959#comment-16860959 ]
Lai Zhou edited comment on SPARK-9983 at 6/11/19 11:51 AM:
-----------------------------------------------------------
[~rxin], `a hyper-optimized single-node version of DataFrame`, do you have any roadmap about it?
In real world, we use spark sql to handle our ETL jobs on Hive. We may extract a lots of user's variables by complex sql queries, which will be the input for machine-learning models.
But when we want to migrate the jobs to real-time system, we always need to interpret these sql queries by another programming language, which requires a lot of work.
Now the local mode of spark sql is not a direct and high performance execution mode, I think it will make great sense to have a hyper-optimized single-node version of DataFrame.
was (Author: hhlai1990):
[~rxin], `a hyper-optimized single-node version of DataFrame`, do you have any roadmap about it?
In real world, we use spark sql to handle our ETL jobs on Hive. We may extract a lots of user's variables by complex sql queries, which will be the input for machine-learning models.
But when we want to migrate the jobs to real-time system, we always need to interpret these sql queries by another programming language,
which requires a lot of work.
Now the local mode of spark sql is not a direct and high performance execution mode, I think it will make great sense to have a high hyper-optimized single-node.
> Local physical operators for query execution
> --------------------------------------------
>
> Key: SPARK-9983
> URL: https://issues.apache.org/jira/browse/SPARK-9983
> Project: Spark
> Issue Type: Story
> Components: SQL
> Reporter: Reynold Xin
> Assignee: Reynold Xin
> Priority: Major
>
> In distributed query execution, there are two kinds of operators:
> (1) operators that exchange data between different executors or threads: examples include broadcast, shuffle.
> (2) operators that process data in a single thread: examples include project, filter, group by, etc.
> This ticket proposes clearly differentiating them and creating local operators in Spark. This leads to a lot of benefits: easier to test, easier to optimize data exchange, better design (single responsibility), and potentially even having a hyper-optimized single-node version of DataFrame.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org