You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Raymond Honderdors <Ra...@sizmek.com> on 2016/04/24 15:50:57 UTC
Spark 2.0 vs 1.6.1 Query Time using Thrift server
Hi,
I run a query using JDBC driver running it on version 1.6.1 it return after 5 - 6 min , the same query against version 2.0 fails after 2h (due to timeout)
Both versions run on the same hardware/configuration and data source
Is anyone else experiencing this?
Also I noticed that the thrift server on version had a change in commands.scala in spark-sql-2.11/execution/command
[cid:image001.png@01D19E48.9F15B950]
The external tool Microstrategy can see tables only with the reinserted code compiling spark version 2.0, it looks like a bug
Raymond Honderdors
Team Lead Analytics BI
Business Intelligence Developer
raymond.honderdors@sizmek.com<ma...@sizmek.com>
T +972.7325.3569
Herzliya
[Read More]<http://feeds.feedburner.com/~r/sizmek-blog/~6/1>
[http://www.sizmek.com/Sizmek.png]<http://www.sizmek.com/>
Re: Spark 2.0 vs 1.6.1 Query Time using Thrift server
Posted by Ted Yu <yu...@gmail.com>.
Spark 2.0 changes very fast.
The ShowTablesCommand is now in:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
Consider opening a PR for the change shown below.
For the query run, can you share some of your code ?
Please pastebin related portion from Thrift server log.
Cheers
On Sun, Apr 24, 2016 at 6:50 AM, Raymond Honderdors <
Raymond.Honderdors@sizmek.com> wrote:
> Hi,
>
>
>
> I run a query using JDBC driver running it on version 1.6.1 it return
> after 5 – 6 min , the same query against version 2.0 fails after 2h (due to
> timeout)
>
> Both versions run on the same hardware/configuration and data source
>
>
>
> Is anyone else experiencing this?
>
>
>
>
>
> Also I noticed that the thrift server on version had a change in
> commands.scala in spark-sql-2.11/execution/command
>
>
>
>
>
>
>
> The external tool Microstrategy can see tables only with the reinserted
> code compiling spark version 2.0, it looks like a bug
>
>
>
>
>
> *Raymond Honderdors *
>
> *Team Lead Analytics BI*
>
> *Business Intelligence Developer *
>
> *raymond.honderdors@sizmek.com <ra...@sizmek.com> *
>
> *T +972.7325.3569*
>
> *Herzliya*
>
>
>
> [image: Read More] <http://feeds.feedburner.com/~r/sizmek-blog/~6/1>
>
> <http://www.sizmek.com/>
>
Re: Spark 2.0 vs 1.6.1 Query Time using Thrift server
Posted by Ted Yu <yu...@gmail.com>.
Spark 2.0 changes very fast.
The ShowTablesCommand is now in:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala
Consider opening a PR for the change shown below.
For the query run, can you share some of your code ?
Please pastebin related portion from Thrift server log.
Cheers
On Sun, Apr 24, 2016 at 6:50 AM, Raymond Honderdors <
Raymond.Honderdors@sizmek.com> wrote:
> Hi,
>
>
>
> I run a query using JDBC driver running it on version 1.6.1 it return
> after 5 – 6 min , the same query against version 2.0 fails after 2h (due to
> timeout)
>
> Both versions run on the same hardware/configuration and data source
>
>
>
> Is anyone else experiencing this?
>
>
>
>
>
> Also I noticed that the thrift server on version had a change in
> commands.scala in spark-sql-2.11/execution/command
>
>
>
>
>
>
>
> The external tool Microstrategy can see tables only with the reinserted
> code compiling spark version 2.0, it looks like a bug
>
>
>
>
>
> *Raymond Honderdors *
>
> *Team Lead Analytics BI*
>
> *Business Intelligence Developer *
>
> *raymond.honderdors@sizmek.com <ra...@sizmek.com> *
>
> *T +972.7325.3569*
>
> *Herzliya*
>
>
>
> [image: Read More] <http://feeds.feedburner.com/~r/sizmek-blog/~6/1>
>
> <http://www.sizmek.com/>
>