You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by soupacabana <ei...@gmail.com> on 2015/02/10 21:32:57 UTC
Spark on very small files, appropriate use case?
Hi all,
I have the following use case:
One job consists of reading from 500-2000 small bzipped logs that are on an
nfs.
(Small means, that the zipped logs are between 0-100KB, average file size is
20KB.)
We read the log lines, do some transformations, and write them to one output
file.
When we do it in pure Python (running the Python script on one core):
-the time for 500 bzipped log files (6.5MB altogether) is about 5 seconds.
-the time for 2000 bzipped log files (25MB altogether) is about 20 seconds.
Because there will be many such jobs, I was thinking of trying Spark for
that purpose.
My preliminary findings and my questions:
*Even only counting the number of log lines with Spark is about 10 times
slower than the entire transformation done by the Python script.
*sc.textfile(list_of_filenames) appear to not perform well on small files,
why?
*sc.wholeTextFiles(path_to_files) performs better than sc.textfile, but does
not support bzipped files. However, also wholeTextFiles does not nearly
provide the speed of the Python script.
*The initialization of a Spark Context takes about 4 seconds. Sending a
Spark job to a cluster takes even longer. Is there a way to decrease this
initialization phase?
*Is my use case actually an appropriate use case for Spark?
Many thanks for your help and comments!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-very-small-files-appropriate-use-case-tp21583.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark on very small files, appropriate use case?
Posted by Kelvin Chu <2d...@gmail.com>.
I had a similar use case before. I found:
1. textFile() produced one partition per file. It can result in many
partitions. I found that calling coalecse() without shuffle helped.
2. If you used persist(), count() will do I/O and put the result into
cache. Transformation later did computation out of the memory cache which
could be much faster.
And, in general, small files hurt I/O performance.
On Tue, Feb 10, 2015 at 12:52 PM, Davies Liu <da...@databricks.com> wrote:
> Spark is an framework to do things in parallel very easy, it
> definitely will help your cases.
>
> def read_file(path):
> lines = open(path).readlines() # bzip2
> return lines
>
> filesRDD = sc.parallelize(path_to_files, N)
> lines = filesRDD.flatMap(read_file)
>
> Then you could do other transforms on lines.
>
> On Tue, Feb 10, 2015 at 12:32 PM, soupacabana <ei...@gmail.com>
> wrote:
> > Hi all,
> >
> > I have the following use case:
> > One job consists of reading from 500-2000 small bzipped logs that are on
> an
> > nfs.
> > (Small means, that the zipped logs are between 0-100KB, average file
> size is
> > 20KB.)
> >
> > We read the log lines, do some transformations, and write them to one
> output
> > file.
> >
> > When we do it in pure Python (running the Python script on one core):
> > -the time for 500 bzipped log files (6.5MB altogether) is about 5
> seconds.
> > -the time for 2000 bzipped log files (25MB altogether) is about 20
> seconds.
> >
> > Because there will be many such jobs, I was thinking of trying Spark for
> > that purpose.
> > My preliminary findings and my questions:
> >
> > *Even only counting the number of log lines with Spark is about 10 times
> > slower than the entire transformation done by the Python script.
> > *sc.textfile(list_of_filenames) appear to not perform well on small
> files,
> > why?
> > *sc.wholeTextFiles(path_to_files) performs better than sc.textfile, but
> does
> > not support bzipped files. However, also wholeTextFiles does not nearly
> > provide the speed of the Python script.
> >
> > *The initialization of a Spark Context takes about 4 seconds. Sending a
> > Spark job to a cluster takes even longer. Is there a way to decrease this
> > initialization phase?
>
> The JVM take about 4 seconds to start up, but a task take only 0.1
> second to start.
>
> > *Is my use case actually an appropriate use case for Spark?
> >
> > Many thanks for your help and comments!
> >
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-very-small-files-appropriate-use-case-tp21583.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark on very small files, appropriate use case?
Posted by Davies Liu <da...@databricks.com>.
Spark is an framework to do things in parallel very easy, it
definitely will help your cases.
def read_file(path):
lines = open(path).readlines() # bzip2
return lines
filesRDD = sc.parallelize(path_to_files, N)
lines = filesRDD.flatMap(read_file)
Then you could do other transforms on lines.
On Tue, Feb 10, 2015 at 12:32 PM, soupacabana <ei...@gmail.com> wrote:
> Hi all,
>
> I have the following use case:
> One job consists of reading from 500-2000 small bzipped logs that are on an
> nfs.
> (Small means, that the zipped logs are between 0-100KB, average file size is
> 20KB.)
>
> We read the log lines, do some transformations, and write them to one output
> file.
>
> When we do it in pure Python (running the Python script on one core):
> -the time for 500 bzipped log files (6.5MB altogether) is about 5 seconds.
> -the time for 2000 bzipped log files (25MB altogether) is about 20 seconds.
>
> Because there will be many such jobs, I was thinking of trying Spark for
> that purpose.
> My preliminary findings and my questions:
>
> *Even only counting the number of log lines with Spark is about 10 times
> slower than the entire transformation done by the Python script.
> *sc.textfile(list_of_filenames) appear to not perform well on small files,
> why?
> *sc.wholeTextFiles(path_to_files) performs better than sc.textfile, but does
> not support bzipped files. However, also wholeTextFiles does not nearly
> provide the speed of the Python script.
>
> *The initialization of a Spark Context takes about 4 seconds. Sending a
> Spark job to a cluster takes even longer. Is there a way to decrease this
> initialization phase?
The JVM take about 4 seconds to start up, but a task take only 0.1
second to start.
> *Is my use case actually an appropriate use case for Spark?
>
> Many thanks for your help and comments!
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-very-small-files-appropriate-use-case-tp21583.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org