You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Wangchangchun (A)" <wa...@huawei.com> on 2015/08/29 05:02:48 UTC

[implement a memory parquet ]

Hi, everyone,
Can somebody help to answer me a question?

In SparkSQL, memory data stored in an object model named internalrow,
If you cache table, spark will convert internalrow into in-memory columnar storage,
We think that in-memory columnar storage is not an efficient storage, we want to try to store it in memory using parquet format.
That is , we want to implement a memory parquet format, and store sparksql cache table in it.

Is this feasible? If it feasible, can someone give me some advise about how to implement it? I should use which parquet APIs?