You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by Geetika Gupta <ge...@knoldus.in> on 2018/05/06 08:58:31 UTC

Issue in loading data in Kudu using Impala

Hi, community

We are trying to load data in kudu using Impala, but we are encountering
the following error:

*Status:* Memory limit exceeded: Error occurred on backend slave3:22000 by
fragment eb410bbc56495c0e:50f46d0800000000 Memory left in process limit:
-438.06 MB Query(eb410bbc56495c0e:50f46d0800000000): Reservation=0
ReservationLimit=30.18 GB OtherMemory=35.95 GB Total=35.95 GB Peak=35.95 GB
Fragment eb410bbc56495c0e:50f46d0800000000: Reservation=0 OtherMemory=35.95
GB Total=35.95 GB Peak=35.95 GB SORT_NODE (id=1): Total=34.62 GB Peak=34.62
GB HDFS_SCAN_NODE (id=0): Total=1.33 GB Peak=1.37 GB KrpcDataStreamSender
(dst_id=2): Total=736.00 B Peak=736.00 B CodeGen: Total=11.72 KB Peak=1.41
MBProcess: memory limit exceeded. Limit=37.72 GB Total=38.15 GB Peak=38.20
GB Buffer Pool: Free Buffers: Total=0 Buffer Pool: Clean Pages: Total=0
Buffer Pool: Unused Reservation: Total=0 Data Stream Service Queue:
Limit=1.89 GB Total=0 Peak=0 Data Stream Manager Early RPCs: Total=0 Peak=0
TCMalloc Overhead: Total=75.40 MB RequestPool=default-pool: Total=35.95 GB
Peak=35.95 GB Query(eb410bbc56495c0e:50f46d0800000000): Reservation=0
ReservationLimit=30.18 GB OtherMemory=35.95 GB Total=35.95 GB Peak=35.95 GB
Untracked Memory: Total=2.13 GBThe quey that we executed is as follows:

*insert into lineitem select * from PARQUETIMPALA500.LINEITEM order by
L_ORDERKEY limit 750000000 offset 536870913*

We are loading data from a parquet table and we are performing the queries
on *500 GB *of *TPCH* data.

We also tried the following query:
*insert into lineitem select * from PARQUETIMPALA500.LINEITEM*

but in this case, the impalad process got aborted after 23hrs of executing
the query.

We are executing the queries on 8 node cluster with the following
configuration
Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz


-- 
Regards,
Geetika Gupta

Issue in loading data in Kudu using Impala

Posted by Geetika Gupta <ge...@knoldus.in>.
Hi, community

We are trying to load data in kudu using Impala, but we are encountering
the following error:

Status: Memory limit exceeded: Error occurred on backend slave3:22000 by
fragment eb410bbc56495c0e:50f46d0800000000 Memory left in process limit:
-438.06 MB Query(eb410bbc56495c0e:50f46d0800000000): Reservation=0
ReservationLimit=30.18 GB OtherMemory=35.95 GB Total=35.95 GB Peak=35.95 GB
Fragment eb410bbc56495c0e:50f46d0800000000: Reservation=0 OtherMemory=35.95
GB Total=35.95 GB Peak=35.95 GB SORT_NODE (id=1): Total=34.62 GB Peak=34.62
GB HDFS_SCAN_NODE (id=0): Total=1.33 GB Peak=1.37 GB KrpcDataStreamSender
(dst_id=2): Total=736.00 B Peak=736.00 B CodeGen: Total=11.72 KB Peak=1.41
MBProcess: memory limit exceeded. Limit=37.72 GB Total=38.15 GB Peak=38.20
GB Buffer Pool: Free Buffers: Total=0 Buffer Pool: Clean Pages: Total=0
Buffer Pool: Unused Reservation: Total=0 Data Stream Service Queue:
Limit=1.89 GB Total=0 Peak=0 Data Stream Manager Early RPCs: Total=0 Peak=0
TCMalloc Overhead: Total=75.40 MB RequestPool=default-pool: Total=35.95 GB
Peak=35.95 GB Query(eb410bbc56495c0e:50f46d0800000000): Reservation=0
ReservationLimit=30.18 GB OtherMemory=35.95 GB Total=35.95 GB Peak=35.95 GB
Untracked Memory: Total=2.13 GB

The quey that we executed is as follows:

* insert** into lineitem select * from PARQUETIMPALA500.LINEITEM order by
L_ORDERKEY limit 750000000 offset 536870913*

We are loading data from a parquet table and we are performing the queries
on* 500 GB* of* TPCH* data.

We also tried the following query:
* insert into lineitem select * from PARQUETIMPALA500.LINEITEM*

but in this case, the impalad process got aborted after 23hrs of executing
the query.

We are executing the queries on 8 node cluster with the following
configuration
Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz

-- 
Regards,
Geetika Gupta