You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Evan Chan <ve...@gmail.com> on 2014/08/20 03:01:14 UTC
Spark SQL: Caching nested structures extremely slow
Hey guys,
I'm using Spark 1.0.2 in AWS with 8 x c3.xlarge machines. I am
working with a subset of the GDELT dataset (57 columns, > 250 million
rows, but my subset is only 4 million) and trying to query it with
Spark SQL.
Since a CSV importer isn't available, my first thought was to use
nested case classes (since Scala has a limit of 22 fields, plus there
are lots of repeated fields in GDELT). The case classes look like
this:
case class ActorInfo(Code: String,
Name: String,
CountryCode: String,
KnownGroupCode: String,
EthnicCode: String, Religion1Code: String,
Religion2Code: String,
Type1Code: String, Type2Code: String, Type3Code: String)
case class GeoInfo(`Type`: Int, FullName: String, CountryCode: String,
ADM1Code: String, Lat: Float,
`Long`: Float, FeatureID: Int)
case class GDeltRow(EventId: Int, Day: Int, MonthYear: Int, Year: Int,
FractionDate: Float,
Actor1: ActorInfo, Actor2: ActorInfo,
IsRootEvent: Byte, EventCode: String, EventBaseCode: String,
EventRootCode: String, QuadClass: Int,
GoldsteinScale: Float,
NumMentions: Int, NumSources: Int, NumArticles: Int,
AvgTone: Float,
Actor1Geo: GeoInfo, Actor2Geo: GeoInfo, ActionGeo:
GeoInfo, DateAdded: String)
Then I use sc.textFile(...) to parse the CSV into an RDD[GDeltRow].
I can query these records without caching. However, if I attempt to
cache using registerAsTable() and then sqlContext.cacheTable(...), it
is extremely slow (takes 1 hour !!).
Any queries using them are also extremely slow.
I had tested Spark SQL using a flat structure (no nesting) on a
different dataset and the caching and queries were both extremely
fast.
Thinking that this is an issue with the case classes, I saved them to
parquet files and used sqlContext.parquetFile(....), but the slowness
is the same. This makes sense, since the internal structure of
SchemaRdds is basically the same. In both cases, for both parquet and
case classes, the schema is the same.
Has anybody else experienced this slowness with nested structures? Is
this a known problem and being worked on?
The only way to work around this issue I can think of is to convert to
JSON, which is tedious, or to construct Parquet files manually (also
tedious).
thanks,
Evan
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark SQL: Caching nested structures extremely slow
Posted by Michael Armbrust <mi...@databricks.com>.
One useful thing to do when you run into unexpected slowness is to run
'jstack' a few times on the driver and executors and see if there is any
particular hotspot in the Spark SQL code.
Also, it seems like a better option here might be to use the new
applySchema API
<https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala#L126>
that has been added for the 1.1 release. I'd be curious how this helps
your performance.
On Thu, Aug 21, 2014 at 1:31 PM, Yin Huai <hu...@gmail.com> wrote:
> I have not profiled this part. But, I think one possible cause is
> allocating an array for every inner struct for every row (every struct
> value is represented by a Spark SQL row). I will play with it later and see
> what I find.
>
>
> On Tue, Aug 19, 2014 at 9:01 PM, Evan Chan <ve...@gmail.com>
> wrote:
>
>> Hey guys,
>>
>> I'm using Spark 1.0.2 in AWS with 8 x c3.xlarge machines. I am
>> working with a subset of the GDELT dataset (57 columns, > 250 million
>> rows, but my subset is only 4 million) and trying to query it with
>> Spark SQL.
>>
>> Since a CSV importer isn't available, my first thought was to use
>> nested case classes (since Scala has a limit of 22 fields, plus there
>> are lots of repeated fields in GDELT). The case classes look like
>> this:
>>
>> case class ActorInfo(Code: String,
>> Name: String,
>> CountryCode: String,
>> KnownGroupCode: String,
>> EthnicCode: String, Religion1Code: String,
>> Religion2Code: String,
>> Type1Code: String, Type2Code: String, Type3Code:
>> String)
>>
>> case class GeoInfo(`Type`: Int, FullName: String, CountryCode: String,
>> ADM1Code: String, Lat: Float,
>> `Long`: Float, FeatureID: Int)
>>
>> case class GDeltRow(EventId: Int, Day: Int, MonthYear: Int, Year: Int,
>> FractionDate: Float,
>> Actor1: ActorInfo, Actor2: ActorInfo,
>> IsRootEvent: Byte, EventCode: String, EventBaseCode:
>> String,
>> EventRootCode: String, QuadClass: Int,
>> GoldsteinScale: Float,
>> NumMentions: Int, NumSources: Int, NumArticles: Int,
>> AvgTone: Float,
>> Actor1Geo: GeoInfo, Actor2Geo: GeoInfo, ActionGeo:
>> GeoInfo, DateAdded: String)
>>
>> Then I use sc.textFile(...) to parse the CSV into an RDD[GDeltRow].
>>
>> I can query these records without caching. However, if I attempt to
>> cache using registerAsTable() and then sqlContext.cacheTable(...), it
>> is extremely slow (takes 1 hour !!).
>>
>> Any queries using them are also extremely slow.
>>
>> I had tested Spark SQL using a flat structure (no nesting) on a
>> different dataset and the caching and queries were both extremely
>> fast.
>>
>> Thinking that this is an issue with the case classes, I saved them to
>> parquet files and used sqlContext.parquetFile(....), but the slowness
>> is the same. This makes sense, since the internal structure of
>> SchemaRdds is basically the same. In both cases, for both parquet and
>> case classes, the schema is the same.
>>
>> Has anybody else experienced this slowness with nested structures? Is
>> this a known problem and being worked on?
>>
>> The only way to work around this issue I can think of is to convert to
>> JSON, which is tedious, or to construct Parquet files manually (also
>> tedious).
>>
>> thanks,
>> Evan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Spark SQL: Caching nested structures extremely slow
Posted by Yin Huai <hu...@gmail.com>.
I have not profiled this part. But, I think one possible cause is
allocating an array for every inner struct for every row (every struct
value is represented by a Spark SQL row). I will play with it later and see
what I find.
On Tue, Aug 19, 2014 at 9:01 PM, Evan Chan <ve...@gmail.com> wrote:
> Hey guys,
>
> I'm using Spark 1.0.2 in AWS with 8 x c3.xlarge machines. I am
> working with a subset of the GDELT dataset (57 columns, > 250 million
> rows, but my subset is only 4 million) and trying to query it with
> Spark SQL.
>
> Since a CSV importer isn't available, my first thought was to use
> nested case classes (since Scala has a limit of 22 fields, plus there
> are lots of repeated fields in GDELT). The case classes look like
> this:
>
> case class ActorInfo(Code: String,
> Name: String,
> CountryCode: String,
> KnownGroupCode: String,
> EthnicCode: String, Religion1Code: String,
> Religion2Code: String,
> Type1Code: String, Type2Code: String, Type3Code:
> String)
>
> case class GeoInfo(`Type`: Int, FullName: String, CountryCode: String,
> ADM1Code: String, Lat: Float,
> `Long`: Float, FeatureID: Int)
>
> case class GDeltRow(EventId: Int, Day: Int, MonthYear: Int, Year: Int,
> FractionDate: Float,
> Actor1: ActorInfo, Actor2: ActorInfo,
> IsRootEvent: Byte, EventCode: String, EventBaseCode:
> String,
> EventRootCode: String, QuadClass: Int,
> GoldsteinScale: Float,
> NumMentions: Int, NumSources: Int, NumArticles: Int,
> AvgTone: Float,
> Actor1Geo: GeoInfo, Actor2Geo: GeoInfo, ActionGeo:
> GeoInfo, DateAdded: String)
>
> Then I use sc.textFile(...) to parse the CSV into an RDD[GDeltRow].
>
> I can query these records without caching. However, if I attempt to
> cache using registerAsTable() and then sqlContext.cacheTable(...), it
> is extremely slow (takes 1 hour !!).
>
> Any queries using them are also extremely slow.
>
> I had tested Spark SQL using a flat structure (no nesting) on a
> different dataset and the caching and queries were both extremely
> fast.
>
> Thinking that this is an issue with the case classes, I saved them to
> parquet files and used sqlContext.parquetFile(....), but the slowness
> is the same. This makes sense, since the internal structure of
> SchemaRdds is basically the same. In both cases, for both parquet and
> case classes, the schema is the same.
>
> Has anybody else experienced this slowness with nested structures? Is
> this a known problem and being worked on?
>
> The only way to work around this issue I can think of is to convert to
> JSON, which is tedious, or to construct Parquet files manually (also
> tedious).
>
> thanks,
> Evan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>