You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sedona.apache.org by 可为 <mw...@qq.com.INVALID> on 2022/09/02 01:46:05 UTC
Why does it take longer to output the result than to computing？
Hello everyone，
I have a question to consult. After making a spatial join of two layers， I output the result using csv format or jdbc.
//csv format
dfContain.write.format("csv").save("file:///F:/data/result")
//jdbc format
dfContain.write
.mode("append")
.option("createTableColumnTypes", "l_gid integer, b_gid integer")
.jdbc(jdbcUrl, "result", prop)
I found it take longer to output the result than to computing. Why? Which method can output faster, except csv or jdbc？ Any sugestions will be appreciated.
Re: Why does it take longer to output the result than to computing？
Posted by Jia Yu <ji...@apache.org>.
This is normal and it is caused by the "lazy evaluation" in Spark. A
spatial join result is only fully materialized when you store it to
external storage or consumed by some other operators (e.g., count).
Otherwise, the query result is not fully materialized and does not finish
the computation. For example, if you run ".show(20)" on a spatial join
query result, it is significantly faster as it only calculates the first 20
rows in the result.
On Thu, Sep 1, 2022 at 8:16 PM 可为 <mw...@qq.com.invalid> wrote:
> Hello everyone，
> I have a question to consult. After making a spatial
> join of two layers， I output the result using csv format or jdbc.
> //csv format
> dfContain.write.format("csv").save("file:///F:/data/result")
> //jdbc format
> dfContain.write
> .mode("append")
> .option("createTableColumnTypes", "l_gid integer, b_gid integer")
> .jdbc(jdbcUrl, "result", prop)
>
>
> I found it take longer to output the result than to computing.
> Why? Which method can output faster, except csv or jdbc？ Any sugestions
> will be appreciated.