You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@carbondata.apache.org by lcxxsg <lc...@163.com> on 2017/10/19 10:14:59 UTC

Failed in insert into carbondata_table select from hive_table

Hi all,
I have trouble with carbondata.
carbondata1.2 + spark2.1

There is a table with more than 300 columns. I create the table like this:

create table carbondata_table_name(
ds String,
event String,
partnercode String,
xx String,
...
)
stored by 'carbondata'
tblproperties(
'DICTIONARY_INCLUDE'='ds, event, partnercode',
'SORT_COLUMNS'='ds, event, partnercode’)

Now,there is data of more than ten million and larger than 60G.I try to
insert them into this table.I set spark properties with
spark.executor.instances=20, spark.executor.cores=1, 
spark.executor.memory=20g
 and spark.driver.memory=20g.
I execute the instruction like "insert into carbondata_table_name select
from hive_table".
But, this job is failed.Message like this:
<http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/file/t29/carbondata_1.png> 
<http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/file/t29/carbondata_2.png> 
<http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/file/t29/carbondata_3.png> 
<http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/file/t29/carbondata_4.png> 
<http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/file/t29/carbondata_5.png> 

anyone know how to solve the problem? Thanks!

Regards



--
Sent from: http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/

Re: Failed in insert into carbondata_table select from hive_table

Posted by Ravindra Pesala <ra...@gmail.com>.

Hi,

It seems your data might contain the data which is longer than the short
limit (> ~ 32000). Currently, carbondata cannot support the column value
more than short limit.

Regards,
Ravindra

On 19 October 2017 at 15:44, lcxxsg <lc...@163.com> wrote:

> Hi all,
> I have trouble with carbondata.
> carbondata1.2 + spark2.1
>
> There is a table with more than 300 columns. I create the table like this:
>
> create table carbondata_table_name(
> ds String,
> event String,
> partnercode String,
> xx String,
> ...
> )
> stored by 'carbondata'
> tblproperties(
> 'DICTIONARY_INCLUDE'='ds, event, partnercode',
> 'SORT_COLUMNS'='ds, event, partnercode’)
>
> Now,there is data of more than ten million and larger than 60G.I try to
> insert them into this table.I set spark properties with
> spark.executor.instances=20, spark.executor.cores=1,
> spark.executor.memory=20g
>  and spark.driver.memory=20g.
> I execute the instruction like "insert into carbondata_table_name select
> from hive_table".
> But, this job is failed.Message like this:
> <http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/file/t29/
> carbondata_1.png>
> <http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/file/t29/
> carbondata_2.png>
> <http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/file/t29/
> carbondata_3.png>
> <http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/file/t29/
> carbondata_4.png>
> <http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/file/t29/
> carbondata_5.png>
>
> anyone know how to solve the problem? Thanks!
>
> Regards
>
>
>
> --
> Sent from: http://apache-carbondata-user-mailing-list.3231.n8.nabble.com/
>



-- 
Thanks & Regards,
Ravi