You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@carbondata.apache.org by "Bo Xu (Jira)" <ji...@apache.org> on 2020/02/12 15:38:00 UTC

[jira] [Created] (CARBONDATA-3695) Apache CarbonData should provides python interface to support deep learning framework PyTorch to ready and write data from/to CarbonData

Bo Xu created CARBONDATA-3695:
---------------------------------

             Summary: Apache CarbonData should provides python interface to support deep learning framework PyTorch to ready and write data from/to CarbonData
                 Key: CARBONDATA-3695
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3695
             Project: CarbonData
          Issue Type: Sub-task
            Reporter: Bo Xu
            Assignee: Bo Xu


Nowadays AI model training is getting more and more popular. Currently many AI framework uses raw data files or row format data files for model training, it could not provide projection, filtering, and fast scan capability like in columnar store. So, if CarbonData supports AI framework, it can speed up model training by increase IO throughput, and provide more flexible training set selection ability to AI developers

AI compute engine integration:

PyTorch integration: New python API in pycarbon to support PyTorch to read data from CarbonData files for training model





--
This message was sent by Atlassian Jira
(v8.3.4#803005)