[DISCUSSION] Presto+Carbon transactional and Non-transactional Write Support

[Akash Nilugal <ak...@gmail.com>]

Hi Community,

As we know the CarbonDataisan indexed columnar data format for fast
analytics on big data platforms. So
we have already integrated with the query engines like spark and even
presto. Currently with presto we
only support the querying of carbon data files. But we don’t yet support
the writing of carbon data files
through presto engine.

Currentlypresto is integrated with carbondata for reading the carbondata
files via presto.
For this, we should be having the store already ready which may be written
carbon in spark and the table
should be hive metastore. So using carbondata connector we are able to read
the carbondata files. But we
cannot create a table or load the data to the table in presto. So it will
somewhat hectic job to read the
carbon files, by writing first with other engines.

So here I will be trying to support the transactional load support in
presto integration for carbon.

I have attached the design document in the Jira, please refer and any
suggestions or input is most welcome.

https://issues.apache.org/jira/browse/CARBONDATA-3831


Regards,
Akash R.

Re: [DISCUSSION] Presto+Carbon transactional and Non-transactional Write Support

[Ajantha Bhat <aj...@gmail.com>]

+ 1 for the proposal,
I didn't see design doc in JIRA. Please check.

Also once we provide write support, it is better to have carbondata as a
separate plugin instead of extending hive.
As presto-hive was not meant to have write support and it is mainly meant
for query at where it exist.

Also hope you cover what all table properties supported in write in the
design doc.

Thanks,
Ajantha




On Thu, May 21, 2020 at 8:57 PM Akash Nilugal <ak...@gmail.com>
wrote:

> Hi Community,
>
> As we know the CarbonDataisan indexed columnar data format for fast
> analytics on big data platforms. So
> we have already integrated with the query engines like spark and even
> presto. Currently with presto we
> only support the querying of carbon data files. But we don’t yet support
> the writing of carbon data files
> through presto engine.
>
> Currentlypresto is integrated with carbondata for reading the carbondata
> files via presto.
> For this, we should be having the store already ready which may be written
> carbon in spark and the table
> should be hive metastore. So using carbondata connector we are able to read
> the carbondata files. But we
> cannot create a table or load the data to the table in presto. So it will
> somewhat hectic job to read the
> carbon files, by writing first with other engines.
>
> So here I will be trying to support the transactional load support in
> presto integration for carbon.
>
> I have attached the design document in the Jira, please refer and any
> suggestions or input is most welcome.
>
> https://issues.apache.org/jira/browse/CARBONDATA-3831
>
>
> Regards,
> Akash R.
>