You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by Liang Chen <ch...@apache.org> on 2023/10/19 07:58:14 UTC
Dear community
As you know, Carbondata as datastore and dataformat already be quite good
and mature.
I want to create the thread via mailing list to open discuss what are the
next milestones of carbondata project？
One proposal from my side: we should consider how to integrate with AI
computing engine？
Regards
Liang
Re: Dear community
Posted by Tao Li <li...@apache.org>.
Hi Liang，
AI technology has broad prospects, and large model technology is in full swing
If we combine AI technology to automatically tune Carbon's parameters, including some predictions, we will become more user-friendly
It's very visionary to exert force here。
On 2023/10/19 07:58:14 Liang Chen wrote:
> As you know, Carbondata as datastore and dataformat already be quite good
> and mature.
> I want to create the thread via mailing list to open discuss what are the
> next milestones of carbondata project？
> One proposal from my side: we should consider how to integrate with AI
> computing engine？
>
> Regards
> Liang
>
Re: Dear community
Posted by Kunal Kapoor <ku...@gmail.com>.
Hey Liang and Xu Bo,
AI seems to be a good direction to move forward.
Ray is also a good option to integrate Carbondata with. It is getting quite
popular and has a strong place in the ML stack.
I suggest upgrading to newer spark versions as they have many good features
for AI/ML.
Also we should upgrade the spark version frequently to leverage these
features.
Another idea that popped into my head is if Carbondata can help in an
offline Feature Store for the model training.
Not sure whether this is feasible or even the right approach, need to
brainstorm on this.
Thanks
Kunal Kapoor
On Thu, 19 Oct, 2023, 7:36 pm Bo Xu, <xu...@apache.org> wrote:
> Agree, CarbonData was focus on bigdata before, and only has very less
> integration with AI, such as PyCarbon, which support PyTorch and TensorFlow
> read data from CarbonData. AI is very popular recently, and has many
> customer need unified data format and storage for bigdata and AI.
> I suggest:
> 1. support developer tools integrade CarbonData, such as jupyter notebook
> and zepplin,
> 2. improve usability of CarbonData, such as support run CarbonData on
> docker and kubernetes easily
> 3.support/enhance different AI framework integrate CarbonData, such as
> TensorFlow/PyTorch/Ray
>
> I hope CarbonData can become unified data format and datastore for
> bigdata,warehouse and AI, User can use the same data with CarbonData in
> different compute engine,such as spark/flink/tensorflow/Pytorch
>
>
> On 2023/10/19 07:58:14 Liang Chen wrote:
> > As you know, Carbondata as datastore and dataformat already be quite good
> > and mature.
> > I want to create the thread via mailing list to open discuss what are the
> > next milestones of carbondata project？
> > One proposal from my side: we should consider how to integrate with AI
> > computing engine？
> >
> > Regards
> > Liang
> >
>
Re: Dear community
Posted by Bo Xu <xu...@apache.org>.
Agree, CarbonData was focus on bigdata before, and only has very less integration with AI, such as PyCarbon, which support PyTorch and TensorFlow read data from CarbonData. AI is very popular recently, and has many customer need unified data format and storage for bigdata and AI.
I suggest:
1. support developer tools integrade CarbonData, such as jupyter notebook and zepplin,
2. improve usability of CarbonData, such as support run CarbonData on docker and kubernetes easily
3.support/enhance different AI framework integrate CarbonData, such as TensorFlow/PyTorch/Ray
I hope CarbonData can become unified data format and datastore for bigdata,warehouse and AI, User can use the same data with CarbonData in different compute engine,such as spark/flink/tensorflow/Pytorch
On 2023/10/19 07:58:14 Liang Chen wrote:
> As you know, Carbondata as datastore and dataformat already be quite good
> and mature.
> I want to create the thread via mailing list to open discuss what are the
> next milestones of carbondata project？
> One proposal from my side: we should consider how to integrate with AI
> computing engine？
>
> Regards
> Liang
>
Re: Dear community
Posted by Indhumathi M <in...@gmail.com>.
Hi Liang,
Agree on your point, to get CarbonData integrated with AI and Machine
Learning, which could help with predictive analytics and also automated
data cleaning.
Some other potential features that we could consider for next roadmap could
be Data versioning, TimeTravel and upgrading Spark, Flink, Presto etc. to
leverage new features.
Regards,
Indhumathi M
On Thu, Oct 19, 2023 at 1:28 PM Liang Chen <ch...@apache.org> wrote:
> As you know, Carbondata as datastore and dataformat already be quite good
> and mature.
> I want to create the thread via mailing list to open discuss what are the
> next milestones of carbondata project？
> One proposal from my side: we should consider how to integrate with AI
> computing engine？
>
> Regards
> Liang
>