You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@doris.apache.org by 陈明雨 <mo...@163.com> on 2021/11/02 08:49:09 UTC
Re:Data Load Issue by using Kettle
We haven't adapted Kettle yet, and the specific implementation of Kettle is to load data line by line by using insert.
This is not an acceptable way for Doris. Doris needs to load data in batches.
It is recommended to use DataX doriswriter plug-in, or mysql odbc external table for mysql data synchronization or load.
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
morningman@163.com;
morningman.cmy@gmail.com
At 2021-10-31 11:29:09, "Thomas Cai" <th...@126.com> wrote:
>Dear All,
>
>
>I tried to load data from MySQL to Doris by using Kettle tool and found there was a limitation of 500 records. It means the job was successful if ETL records were less than 500. And the job failed when the records were more than 500, but the 500 records had still been loaded into Doris.
>
>
>Does anyone meet the same situation by using Kettle before? Or there is any Doris variance setting to fix the issue? Thanks.
>
>
>Best Regards,
>Thomas
Re:Re: Data Load Issue by using Kettle
Posted by Thomas Cai <th...@126.com>.
Hi Matt, Thanks for your recommendation.
Regards,
Thomas
At 2021-11-03 07:30:30, "Matt Casters" <ma...@neotechnology.com.INVALID> wrote:
>>> It is recommended to use DataX doriswriter plug-in
>
>While I'm no longer involved in Kettle we have a number of bulk loaders in
>Apache Hop and would be happy to work with you in figuring out how to
>quickly load data into Doris.
>
>Op di 2 nov. 2021 09:49 schreef 陈明雨 <mo...@163.com>:
>
>> We haven't adapted Kettle yet, and the specific implementation of Kettle
>> is to load data line by line by using insert.
>> This is not an acceptable way for Doris. Doris needs to load data in
>> batches.
>>
>> It is recommended to use DataX doriswriter plug-in, or mysql odbc external
>> table for mysql data synchronization or load.
>>
>>
>>
>>
>> --
>>
>> 此致！Best Regards
>> 陈明雨 Mingyu Chen
>>
>> Email:
>> morningman@163.com;
>> morningman.cmy@gmail.com
>>
>>
>>
>>
>>
>> At 2021-10-31 11:29:09, "Thomas Cai" <th...@126.com> wrote:
>> >Dear All,
>> >
>> >
>> >I tried to load data from MySQL to Doris by using Kettle tool and found
>> there was a limitation of 500 records. It means the job was successful if
>> ETL records were less than 500. And the job failed when the records were
>> more than 500, but the 500 records had still been loaded into Doris.
>> >
>> >
>> >Does anyone meet the same situation by using Kettle before? Or there is
>> any Doris variance setting to fix the issue? Thanks.
>> >
>> >
>> >Best Regards,
>> >Thomas
>>
Re: Data Load Issue by using Kettle
Posted by Matt Casters <ma...@neotechnology.com.INVALID>.
>> It is recommended to use DataX doriswriter plug-in
While I'm no longer involved in Kettle we have a number of bulk loaders in
Apache Hop and would be happy to work with you in figuring out how to
quickly load data into Doris.
Op di 2 nov. 2021 09:49 schreef 陈明雨 <mo...@163.com>:
> We haven't adapted Kettle yet, and the specific implementation of Kettle
> is to load data line by line by using insert.
> This is not an acceptable way for Doris. Doris needs to load data in
> batches.
>
> It is recommended to use DataX doriswriter plug-in, or mysql odbc external
> table for mysql data synchronization or load.
>
>
>
>
> --
>
> 此致！Best Regards
> 陈明雨 Mingyu Chen
>
> Email:
> morningman@163.com;
> morningman.cmy@gmail.com
>
>
>
>
>
> At 2021-10-31 11:29:09, "Thomas Cai" <th...@126.com> wrote:
> >Dear All,
> >
> >
> >I tried to load data from MySQL to Doris by using Kettle tool and found
> there was a limitation of 500 records. It means the job was successful if
> ETL records were less than 500. And the job failed when the records were
> more than 500, but the 500 records had still been loaded into Doris.
> >
> >
> >Does anyone meet the same situation by using Kettle before? Or there is
> any Doris variance setting to fix the issue? Thanks.
> >
> >
> >Best Regards,
> >Thomas
>
Re: Data Load Issue by using Kettle
Posted by Matt Casters <ma...@neotechnology.com.INVALID>.
>> It is recommended to use DataX doriswriter plug-in
While I'm no longer involved in Kettle we have a number of bulk loaders in
Apache Hop and would be happy to work with you in figuring out how to
quickly load data into Doris.
Op di 2 nov. 2021 09:49 schreef 陈明雨 <mo...@163.com>:
> We haven't adapted Kettle yet, and the specific implementation of Kettle
> is to load data line by line by using insert.
> This is not an acceptable way for Doris. Doris needs to load data in
> batches.
>
> It is recommended to use DataX doriswriter plug-in, or mysql odbc external
> table for mysql data synchronization or load.
>
>
>
>
> --
>
> 此致！Best Regards
> 陈明雨 Mingyu Chen
>
> Email:
> morningman@163.com;
> morningman.cmy@gmail.com
>
>
>
>
>
> At 2021-10-31 11:29:09, "Thomas Cai" <th...@126.com> wrote:
> >Dear All,
> >
> >
> >I tried to load data from MySQL to Doris by using Kettle tool and found
> there was a limitation of 500 records. It means the job was successful if
> ETL records were less than 500. And the job failed when the records were
> more than 500, but the 500 records had still been loaded into Doris.
> >
> >
> >Does anyone meet the same situation by using Kettle before? Or there is
> any Doris variance setting to fix the issue? Thanks.
> >
> >
> >Best Regards,
> >Thomas
>
Re:Re:Data Load Issue by using Kettle
Posted by Thomas Cai <th...@126.com>.
MingYu,
Thanks for your recommendation.
Regards,
Thomas
At 2021-11-02 16:49:09, "陈明雨" <mo...@163.com> wrote:
>We haven't adapted Kettle yet, and the specific implementation of Kettle is to load data line by line by using insert.
>This is not an acceptable way for Doris. Doris needs to load data in batches.
>
>It is recommended to use DataX doriswriter plug-in, or mysql odbc external table for mysql data synchronization or load.
>
>
>
>
>--
>
>此致！Best Regards
>陈明雨 Mingyu Chen
>
>Email:
>morningman@163.com;
>morningman.cmy@gmail.com
>
>
>
>
>
>At 2021-10-31 11:29:09, "Thomas Cai" <th...@126.com> wrote:
>>Dear All,
>>
>>
>>I tried to load data from MySQL to Doris by using Kettle tool and found there was a limitation of 500 records. It means the job was successful if ETL records were less than 500. And the job failed when the records were more than 500, but the 500 records had still been loaded into Doris.
>>
>>
>>Does anyone meet the same situation by using Kettle before? Or there is any Doris variance setting to fix the issue? Thanks.
>>
>>
>>Best Regards,
>>Thomas