Alternatives to HDFS for Storage Layer

["Adunuthula, Seshu" <sa...@ebay.com>]

Hello Folks,

Spent some time over the weekend looking at Apache Carbon Data. This looks quite interesting, specifically the deep integration with SparkSQL. Huawei contributions have been impressive (CBO for SparkSQL and Apache CarbonData). You have all the pieces to make a credible alternative to native DW solutions such as Terdata/Vertica/Netezza.

We at eBay plan to do a POC for interactive users with SparkSQL + CarbonData and will let you know our results. Curious about your thoughts on alternatives to HDFS for the Storage layer to Carbon Data Files. For e.g. If you store these files into a KV Store (Mongo/Couchbase). Would it make a difference to the raw performance for the queries?

Regards
Seshu Adunuthula

Re: Alternatives to HDFS for Storage Layer

[Ravindra Pesala <ra...@gmail.com>]

Hi ,

Thanks for analyzing CarbonData. Currently, we are supporting only HDFS as
storage and soon we support S3 as well. And yes we have a long goal to
support our own storage but still not yet realized any design for the same.

Regards,
Ravindra.

On 10 October 2017 at 02:12, Adunuthula, Seshu <sa...@ebay.com> wrote:

> Hello Folks,
>
> Spent some time over the weekend looking at Apache Carbon Data. This looks
> quite interesting, specifically the deep integration with SparkSQL. Huawei
> contributions have been impressive (CBO for SparkSQL and Apache
> CarbonData). You have all the pieces to make a credible alternative to
> native DW solutions such as Terdata/Vertica/Netezza.
>
> We at eBay plan to do a POC for interactive users with SparkSQL +
> CarbonData and will let you know our results. Curious about your thoughts
> on alternatives to HDFS for the Storage layer to Carbon Data Files. For
> e.g. If you store these files into a KV Store (Mongo/Couchbase). Would it
> make a difference to the raw performance for the queries?
>
> Regards
> Seshu Adunuthula
>
>


-- 
Thanks & Regards,
Ravi