You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Exie <tf...@prodevelop.com.au> on 2015/06/30 07:14:45 UTC

Spark 1.4.0: read.df() causes excessive IO

Hi Folks,

I just stepped up from 1.3.1 to 1.4.0, the most notable difference for me so
far is the data frame reader/writer. Previously:
   val myData = hiveContext.load("s3n://someBucket/somePath/","parquet")
Now:
   val myData = hiveContext.read.parquet("s3n://someBucket/somePath")

Using the original code, it didnt actually fire an action, but using the
data frame reader, it does.

So if I have say, 1Gb of data in a parquet file partitioned into 1000
chunks, I now have to sit there waiting for it to scan that 1000 chunks.

It seems to be doing some sort of validation ? is there any way to switch
that off or stop it hitting the source so much ?

I should also note the parquet files were written with 1.6RC3, where as I
think Spark 1.4.0 is using Parquet 1.7.x



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-0-read-df-causes-excessive-IO-tp23541.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Spark 1.4.0: read.df() causes excessive IO

Posted by Exie <tf...@prodevelop.com.au>.
Just to add to this, here's some more info:

val myDF = hiveContext.read.parquet("s3n://myBucket/myPath/")

Produces these...
2015-07-01 03:25:50,450  INFO [pool-14-thread-4]
(org.apache.hadoop.fs.s3native.NativeS3FileSystem) - Opening
's3n://myBucket/myPath/part-r-00339.parquet' for reading

That is to say, it actually opens and reads every frick'n file.
Previously, it would have queued the command until an action was called.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-4-0-read-df-causes-excessive-IO-tp23541p23559.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org