You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by lu...@sina.com on 2016/08/09 09:28:10 UTC

回复：saving DF to HDFS in parquet format very slow in SparkSQL app

maybe this problem is not so easy to understand, so I attached my full code.
Hope this could help in solving the problem.

--------------------------------

 

Thanks&Best regards!
San.Luo

----- 原始邮件 -----
发件人：<lu...@sina.com>
收件人："user" <us...@spark.apache.org>
主题：saving DF to HDFS in parquet format very slow in SparkSQL app
日期：2016年08月09日 15点34分

hi there:    I got a problem in saving a DF to HDFS as parquet format very slow. And I attached a pic which shows a lot of time is spent in getting result.the code is :streamingData.write.mode(SaveMode.Overwrite).parquet("/data/streamingData")
I don't quite understand why my app is so slow in getting the result. I tried to access my HDFS while the app is running slow , HDFS is ok.
Any idea will be appreciated.


--------------------------------

 

Thanks&amp;Best regards!
San.Luo