You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kudu.apache.org by sky <x_...@163.com> on 2017/08/22 08:06:39 UTC
KUDU INSERT SLOWLY
Hi all,
I do kudu insert sql through the impala shell, it is very slowly.Approximately 4 minutes two thousand insert sqls. Is there any way to speed up kudu insert speed?
Command：
./impala-shell.sh -f a.sql
a.sql:
insert into table t1 values("a1","b");
insert into table t1 valuse("a2","b");
......
Re: KUDU INSERT SLOWLY
Posted by Matthew Jacobs <mj...@cloudera.com>.
There are some improvements we hope to make in Impala to reduce the
planning overhead for these simple DML statements. You can track that
work with: https://issues.apache.org/jira/browse/IMPALA-5277
-Matt
On Tue, Aug 22, 2017 at 1:12 AM, Balázs Jeszenszky <ba...@cloudera.com> wrote:
> [dev@ to bcc]
>
> Hello,
>
> Setting up a query has some overhead (in query planning). I expect it
> to be in the 100 ms range, multiplying that by 2k adds up to the time
> you report.
> Inserting all the values through one query would help.
>
> HTH
>
> On Tue, Aug 22, 2017 at 10:06 AM, sky <x_...@163.com> wrote:
>> Hi all,
>> I do kudu insert sql through the impala shell, it is very slowly.Approximately 4 minutes two thousand insert sqls. Is there any way to speed up kudu insert speed?
>> Command：
>> ./impala-shell.sh -f a.sql
>> a.sql:
>> insert into table t1 values("a1","b");
>> insert into table t1 valuse("a2","b");
>> ......
>
>
>
> --
> Balázs Jeszenszky
> Customer Operations Engineer
> Email: balazsj@cloudera.com
> Data: http://www.cloudera.com
Re: Re: KUDU INSERT SLOWLY
Posted by Jean-Daniel Cryans <jd...@apache.org>.
(putting dev@ in bcc again, please be mindful of which mailing list you
write to)
The best way would be to follow the "Batch insert" method described here:
http://kudu.apache.org/docs/kudu_impala_integration.html#kudu_impala_insert_bulk
J-D
On Wed, Aug 23, 2017 at 6:30 PM, sky <x_...@163.com> wrote:
> Thank you for your method.
> Is there any other way to quickly load text files into kudu table ?
>
>
>
>
>
>
>
>
> At 2017-08-22 16:12:47, "Balázs Jeszenszky" <ba...@cloudera.com> wrote:
> >[dev@ to bcc]
> >
> >Hello,
> >
> >Setting up a query has some overhead (in query planning). I expect it
> >to be in the 100 ms range, multiplying that by 2k adds up to the time
> >you report.
> >Inserting all the values through one query would help.
> >
> >HTH
> >
> >On Tue, Aug 22, 2017 at 10:06 AM, sky <x_...@163.com> wrote:
> >> Hi all,
> >> I do kudu insert sql through the impala shell, it is very
> slowly.Approximately 4 minutes two thousand insert sqls. Is there any way
> to speed up kudu insert speed?
> >> Command：
> >> ./impala-shell.sh -f a.sql
> >> a.sql:
> >> insert into table t1 values("a1","b");
> >> insert into table t1 valuse("a2","b");
> >> ......
> >
> >
> >
> >--
> >Balázs Jeszenszky
> >Customer Operations Engineer
> >Email: balazsj@cloudera.com
> >Data: http://www.cloudera.com
>
Re: Re: KUDU INSERT SLOWLY
Posted by Jean-Daniel Cryans <jd...@apache.org>.
(putting dev@ in bcc again, please be mindful of which mailing list you
write to)
The best way would be to follow the "Batch insert" method described here:
http://kudu.apache.org/docs/kudu_impala_integration.html#kudu_impala_insert_bulk
J-D
On Wed, Aug 23, 2017 at 6:30 PM, sky <x_...@163.com> wrote:
> Thank you for your method.
> Is there any other way to quickly load text files into kudu table ?
>
>
>
>
>
>
>
>
> At 2017-08-22 16:12:47, "Balázs Jeszenszky" <ba...@cloudera.com> wrote:
> >[dev@ to bcc]
> >
> >Hello,
> >
> >Setting up a query has some overhead (in query planning). I expect it
> >to be in the 100 ms range, multiplying that by 2k adds up to the time
> >you report.
> >Inserting all the values through one query would help.
> >
> >HTH
> >
> >On Tue, Aug 22, 2017 at 10:06 AM, sky <x_...@163.com> wrote:
> >> Hi all,
> >> I do kudu insert sql through the impala shell, it is very
> slowly.Approximately 4 minutes two thousand insert sqls. Is there any way
> to speed up kudu insert speed?
> >> Command：
> >> ./impala-shell.sh -f a.sql
> >> a.sql:
> >> insert into table t1 values("a1","b");
> >> insert into table t1 valuse("a2","b");
> >> ......
> >
> >
> >
> >--
> >Balázs Jeszenszky
> >Customer Operations Engineer
> >Email: balazsj@cloudera.com
> >Data: http://www.cloudera.com
>
Re:Re: KUDU INSERT SLOWLY
Posted by sky <x_...@163.com>.
Thank you for your method.
Is there any other way to quickly load text files into kudu table ?
At 2017-08-22 16:12:47, "Balázs Jeszenszky" <ba...@cloudera.com> wrote:
>[dev@ to bcc]
>
>Hello,
>
>Setting up a query has some overhead (in query planning). I expect it
>to be in the 100 ms range, multiplying that by 2k adds up to the time
>you report.
>Inserting all the values through one query would help.
>
>HTH
>
>On Tue, Aug 22, 2017 at 10:06 AM, sky <x_...@163.com> wrote:
>> Hi all,
>> I do kudu insert sql through the impala shell, it is very slowly.Approximately 4 minutes two thousand insert sqls. Is there any way to speed up kudu insert speed?
>> Command：
>> ./impala-shell.sh -f a.sql
>> a.sql:
>> insert into table t1 values("a1","b");
>> insert into table t1 valuse("a2","b");
>> ......
>
>
>
>--
>Balázs Jeszenszky
>Customer Operations Engineer
>Email: balazsj@cloudera.com
>Data: http://www.cloudera.com
Re: KUDU INSERT SLOWLY
Posted by Balázs Jeszenszky <ba...@cloudera.com>.
[dev@ to bcc]
Hello,
Setting up a query has some overhead (in query planning). I expect it
to be in the 100 ms range, multiplying that by 2k adds up to the time
you report.
Inserting all the values through one query would help.
HTH
On Tue, Aug 22, 2017 at 10:06 AM, sky <x_...@163.com> wrote:
> Hi all,
> I do kudu insert sql through the impala shell, it is very slowly.Approximately 4 minutes two thousand insert sqls. Is there any way to speed up kudu insert speed?
> Command：
> ./impala-shell.sh -f a.sql
> a.sql:
> insert into table t1 values("a1","b");
> insert into table t1 valuse("a2","b");
> ......
--
Balázs Jeszenszky
Customer Operations Engineer
Email: balazsj@cloudera.com
Data: http://www.cloudera.com
Re: KUDU INSERT SLOWLY
Posted by Balázs Jeszenszky <ba...@cloudera.com>.
[dev@ to bcc]
Hello,
Setting up a query has some overhead (in query planning). I expect it
to be in the 100 ms range, multiplying that by 2k adds up to the time
you report.
Inserting all the values through one query would help.
HTH
On Tue, Aug 22, 2017 at 10:06 AM, sky <x_...@163.com> wrote:
> Hi all,
> I do kudu insert sql through the impala shell, it is very slowly.Approximately 4 minutes two thousand insert sqls. Is there any way to speed up kudu insert speed?
> Command：
> ./impala-shell.sh -f a.sql
> a.sql:
> insert into table t1 values("a1","b");
> insert into table t1 valuse("a2","b");
> ......
--
Balázs Jeszenszky
Customer Operations Engineer
Email: balazsj@cloudera.com
Data: http://www.cloudera.com