You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by Geetika Gupta <ge...@knoldus.in> on 2018/05/07 05:42:04 UTC
Issue in data loading in Impala + Kudu
Hi community,
I was trying to load 500GB of TPCH data into kudu table using the following
query:
*insert into lineitem select * from PARQUETIMPALA500.LINEITEM*
While executing the query for around 17 hrs it got cancelled as the impalad
process of that machine got aborted. Here are the logs of the impalad
process.
*impalad.ERROR*
Log file created at: 2018/05/06 13:40:34
Running on machine: slave2
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to this
file.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/root/softwares/impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/root/softwares/impala/testdata/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The short-circuit
local reads feature cannot be used because libhadoop cannot be loaded.
tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
0x7fd9e9fc3929
tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176
0x7fd9e9fc3929
F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
log_entry.count > 0 (-1831809966 vs. 0)
*** Check failure stack trace: ***
@ 0x3fc0c0d google::LogMessage::Fail()
@ 0x3fc24b2 google::LogMessage::SendToLog()
@ 0x3fc05e7 google::LogMessage::Flush()
@ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal()
@ 0x1bbcb31 impala::PrintErrorMap()
@ 0x1bbcd07 impala::PrintErrorMapToString()
@ 0x2decbd7 impala::Coordinator::GetErrorLog()
@ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
@ 0x1b29264 impala::ImpalaServer::CloseOperation()
@ 0x2c5ce86
apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseOperation()
@ 0x2c56b8c
apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
@ 0x2c2fcb1
impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
@ 0x16fdb20 apache::thrift::TDispatchProcessor::process()
@ 0x18ea6b3
apache::thrift::server::TAcceptQueueServer::Task::run()
@ 0x18e2181 impala::ThriftThread::RunRunnable()
@ 0x18e3885 boost::_mfi::mf2<>::operator()()
@ 0x18e371b boost::_bi::list3<>::operator()<>()
@ 0x18e3467 boost::_bi::bind_t<>::operator()()
@ 0x18e337a
boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x192761c boost::function0<>::operator()()
@ 0x1c3ebf7 impala::Thread::SuperviseThread()
@ 0x1c470cd boost::_bi::list5<>::operator()<>()
@ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
@ 0x1c46fb4 boost::detail::thread_data<>::run()
@ 0x2eedb4a thread_proxy
@ 0x7fda1dbb16ba start_thread
@ 0x7fda1d8e741d clone
Wrote minidump to
/tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
*impalad.FATAL*
Log file created at: 2018/05/07 09:46:12
Running on machine: slave2
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
log_entry.count > 0 (-1831809966 vs. 0)
*Impalad.INFO*
edentials={real_user=root}} blocked reactor thread for 34288.6us
I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC call
kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
(slave5), user_credentials={real_user=root}} blocked reactor thread for
35859.8us
I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC call
kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
(slave5), user_credentials={real_user=root}} blocked reactor thread for
40664.9us
I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC call
kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
(slave5), user_credentials={real_user=root}} blocked reactor thread for
49514.6us
I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission control
resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
log_entry.count > 0 (-1831809966 vs. 0)
Wrote minidump to
/tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
*Note*:
We are executing the queries on 8 node cluster with the following
configuration
Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
--
Regards,
Geetika Gupta
Re: 答复: Issue in data loading in Impala + Kudu
Posted by Todd Lipcon <to...@cloudera.com>.
What version of Kudu?
Also, just one disk on these nodes? It sounds like it is not able to keep
up flushing at the rate you are inserting, and then memory is filling up. I
would double check that your disks are appropriate for the workload.
-Todd
On Sat, May 12, 2018 at 1:46 AM, Geetika Gupta <ge...@knoldus.in>
wrote:
> Hi community,
>
> We were trying to load 500GB of TPCH data in the lineitem table using the
> following query:
>
> insert into LINEITEM select L_ORDERKEY, L_LINENUMBER, L_PARTKEY,
> L_SUPPKEY, L_SHIPDATE,L_RECEIPTDATE, L_SHIPMODE,
> L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT, L_TAX, L_RETURNFLAG,L_LINESTATUS,
> L_COMMITDATE,L_SHIPINSTRUCT, L_COMMENT from PARQUETIMPALA500.LINEITEM
>
> but the query gives us the following exception:
>
> *Status:* Kudu error(s) reported, first error: Timed out: Failed to write
> batch of 51973 ops to tablet 2b1e63c335b646f2859ba583d736f109 after 337
> attempt(s): Failed to write to server: (no server available): Write(tablet:
> 2b1e63c335b646f2859ba583d736f109, num_ops: 51973, num_attempts: 337)
> passed its deadline: Remote error: Service unavailable: Soft memory limit
> exceeded (at 99.66% of capacity)We are using the default configuration
> properties for KUDU. The values for some configuration parameters is as
> follows:
> --memory_limit_soft_percentage=80
> --memory_limit_hard_bytes=0
>
> We are executing the queries on an impala cluster. Below are the
> configuration of the nodes:
>
> Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
> Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>
> We also looked into the tablet servers tab in the kudu master UI, all the
> tablet servers were active there, could not figure out what was the actual
> reason for the exception.
>
> On Mon, May 7, 2018 at 11:30 AM, helifu <hz...@corp.netease.com> wrote:
>
>> Hi Geetika,
>>
>> It would be better to ask this question in impala user mail list. Here is
>> the impala community: https://impala.apache.org/community.html
>>
>>
>>
>> 何李夫
>>
>> 2018-05-07 13:56:02
>>
>>
>>
>> *发件人:* user-return-1353-hzhelifu=corp.netease.com@kudu.apache.org
>> <us...@kudu.apache.org> *代表 *Geetika
>> Gupta
>> *发送时间:* 2018年5月7日 13:42
>> *收件人:* user@kudu.apache.org
>> *主题:* Issue in data loading in Impala + Kudu
>>
>>
>>
>> Hi community,
>>
>>
>>
>> I was trying to load 500GB of TPCH data into kudu table using the
>> following query:
>>
>>
>>
>> *insert into lineitem select * from PARQUETIMPALA500.LINEITEM*
>>
>>
>> While executing the query for around 17 hrs it got cancelled as the
>> impalad process of that machine got aborted. Here are the logs of the
>> impalad process.
>>
>>
>>
>> *impalad.ERROR*
>>
>>
>>
>> Log file created at: 2018/05/06 13:40:34
>>
>> Running on machine: slave2
>>
>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>
>> E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to this
>> file.
>>
>> SLF4J: Class path contains multiple SLF4J bindings.
>>
>> SLF4J: Found binding in [jar:file:/root/softwares/impa
>> la/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/
>> impl/StaticLoggerBinder.class]
>>
>> SLF4J: Found binding in [jar:file:/root/softwares/impa
>> la/testdata/target/dependency/slf4j-log4j12-1.7.25.jar!/org/
>> slf4j/impl/StaticLoggerBinder.class]
>>
>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> explanation.
>>
>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>>
>> 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>>
>> 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
>> short-circuit local reads feature cannot be used because libhadoop cannot
>> be loaded.
>>
>> tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
>> 0x7fd9e9fc3929
>>
>> tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176
>> 0x7fd9e9fc3929
>>
>> F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> log_entry.count > 0 (-1831809966 vs. 0)
>>
>> *** Check failure stack trace: ***
>>
>> @ 0x3fc0c0d google::LogMessage::Fail()
>>
>> @ 0x3fc24b2 google::LogMessage::SendToLog()
>>
>> @ 0x3fc05e7 google::LogMessage::Flush()
>>
>> @ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal()
>>
>> @ 0x1bbcb31 impala::PrintErrorMap()
>>
>> @ 0x1bbcd07 impala::PrintErrorMapToString()
>>
>> @ 0x2decbd7 impala::Coordinator::GetErrorLog()
>>
>> @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
>>
>> @ 0x1b29264 impala::ImpalaServer::CloseOperation()
>>
>> @ 0x2c5ce86 apache::hive::service::cli::th
>> rift::TCLIServiceProcessor::process_CloseOperation()
>>
>> @ 0x2c56b8c apache::hive::service::cli::th
>> rift::TCLIServiceProcessor::dispatchCall()
>>
>> @ 0x2c2fcb1 impala::ImpalaHiveServer2Servi
>> ceProcessor::dispatchCall()
>>
>> @ 0x16fdb20 apache::thrift::TDispatchProcessor::process()
>>
>> @ 0x18ea6b3 apache::thrift::server::TAccep
>> tQueueServer::Task::run()
>>
>> @ 0x18e2181 impala::ThriftThread::RunRunnable()
>>
>> @ 0x18e3885 boost::_mfi::mf2<>::operator()()
>>
>> @ 0x18e371b boost::_bi::list3<>::operator()<>()
>>
>> @ 0x18e3467 boost::_bi::bind_t<>::operator()()
>>
>> @ 0x18e337a boost::detail::function::void_
>> function_obj_invoker0<>::invoke()
>>
>> @ 0x192761c boost::function0<>::operator()()
>>
>> @ 0x1c3ebf7 impala::Thread::SuperviseThread()
>>
>> @ 0x1c470cd boost::_bi::list5<>::operator()<>()
>>
>> @ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
>>
>> @ 0x1c46fb4 boost::detail::thread_data<>::run()
>>
>> @ 0x2eedb4a thread_proxy
>>
>> @ 0x7fda1dbb16ba start_thread
>>
>> @ 0x7fda1d8e741d clone
>>
>> Wrote minidump to /tmp/minidumps/impalad/a9113d9
>> b-bc3d-488a-1feebf9b-47b42022.dmp
>>
>>
>>
>> *impalad.FATAL*
>>
>>
>>
>> Log file created at: 2018/05/07 09:46:12
>>
>> Running on machine: slave2
>>
>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>>
>> F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> log_entry.count > 0 (-1831809966 vs. 0)
>>
>>
>>
>> *Impalad.INFO*
>>
>> edentials={real_user=root}} blocked reactor thread for 34288.6us
>>
>> I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> (slave5), user_credentials={real_user=root}} blocked reactor thread for
>> 35859.8us
>>
>> I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> (slave5), user_credentials={real_user=root}} blocked reactor thread for
>> 40664.9us
>>
>> I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
>> (slave5), user_credentials={real_user=root}} blocked reactor thread for
>> 49514.6us
>>
>> I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission control
>> resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
>>
>> F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
>> log_entry.count > 0 (-1831809966 vs. 0)
>>
>> Wrote minidump to /tmp/minidumps/impalad/a9113d9
>> b-bc3d-488a-1feebf9b-47b42022.dmp
>>
>>
>>
>> *Note*:
>>
>> We are executing the queries on 8 node cluster with the following
>> configuration
>>
>> Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
>> Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>>
>>
>>
>>
>>
>> --
>>
>> Regards,
>>
>> Geetika Gupta
>>
>
>
>
> --
> Regards,
> Geetika Gupta
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: 答复: Issue in data loading in Impala + Kudu
Posted by Geetika Gupta <ge...@knoldus.in>.
Hi community,
We were trying to load 500GB of TPCH data in the lineitem table using the
following query:
insert into LINEITEM select L_ORDERKEY, L_LINENUMBER, L_PARTKEY, L_SUPPKEY,
L_SHIPDATE,L_RECEIPTDATE, L_SHIPMODE,
L_QUANTITY,L_EXTENDEDPRICE,L_DISCOUNT, L_TAX, L_RETURNFLAG,L_LINESTATUS,
L_COMMITDATE,L_SHIPINSTRUCT, L_COMMENT from PARQUETIMPALA500.LINEITEM
but the query gives us the following exception:
*Status:* Kudu error(s) reported, first error: Timed out: Failed to write
batch of 51973 ops to tablet 2b1e63c335b646f2859ba583d736f109 after 337
attempt(s): Failed to write to server: (no server available): Write(tablet:
2b1e63c335b646f2859ba583d736f109, num_ops: 51973, num_attempts: 337) passed
its deadline: Remote error: Service unavailable: Soft memory limit exceeded
(at 99.66% of capacity)We are using the default configuration properties
for KUDU. The values for some configuration parameters is as follows:
--memory_limit_soft_percentage=80
--memory_limit_hard_bytes=0
We are executing the queries on an impala cluster. Below are the
configuration of the nodes:
Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
We also looked into the tablet servers tab in the kudu master UI, all the
tablet servers were active there, could not figure out what was the actual
reason for the exception.
On Mon, May 7, 2018 at 11:30 AM, helifu <hz...@corp.netease.com> wrote:
> Hi Geetika,
>
> It would be better to ask this question in impala user mail list. Here is
> the impala community: https://impala.apache.org/community.html
>
>
>
> 何李夫
>
> 2018-05-07 13:56:02
>
>
>
> *发件人:* user-return-1353-hzhelifu=corp.netease.com@kudu.apache.org
> <us...@kudu.apache.org> *代表 *Geetika
> Gupta
> *发送时间:* 2018年5月7日 13:42
> *收件人:* user@kudu.apache.org
> *主题:* Issue in data loading in Impala + Kudu
>
>
>
> Hi community,
>
>
>
> I was trying to load 500GB of TPCH data into kudu table using the
> following query:
>
>
>
> *insert into lineitem select * from PARQUETIMPALA500.LINEITEM*
>
>
> While executing the query for around 17 hrs it got cancelled as the
> impalad process of that machine got aborted. Here are the logs of the
> impalad process.
>
>
>
> *impalad.ERROR*
>
>
>
> Log file created at: 2018/05/06 13:40:34
>
> Running on machine: slave2
>
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>
> E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to this
> file.
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [jar:file:/root/softwares/
> impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/
> slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in [jar:file:/root/softwares/impala/testdata/target/
> dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The
> short-circuit local reads feature cannot be used because libhadoop cannot
> be loaded.
>
> tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176
> 0x7fd9e9fc3929
>
> tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176
> 0x7fd9e9fc3929
>
> F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
> log_entry.count > 0 (-1831809966 vs. 0)
>
> *** Check failure stack trace: ***
>
> @ 0x3fc0c0d google::LogMessage::Fail()
>
> @ 0x3fc24b2 google::LogMessage::SendToLog()
>
> @ 0x3fc05e7 google::LogMessage::Flush()
>
> @ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal()
>
> @ 0x1bbcb31 impala::PrintErrorMap()
>
> @ 0x1bbcd07 impala::PrintErrorMapToString()
>
> @ 0x2decbd7 impala::Coordinator::GetErrorLog()
>
> @ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
>
> @ 0x1b29264 impala::ImpalaServer::CloseOperation()
>
> @ 0x2c5ce86 apache::hive::service::cli::
> thrift::TCLIServiceProcessor::process_CloseOperation()
>
> @ 0x2c56b8c apache::hive::service::cli::
> thrift::TCLIServiceProcessor::dispatchCall()
>
> @ 0x2c2fcb1 impala::ImpalaHiveServer2ServiceProces
> sor::dispatchCall()
>
> @ 0x16fdb20 apache::thrift::TDispatchProcessor::process()
>
> @ 0x18ea6b3 apache::thrift::server::
> TAcceptQueueServer::Task::run()
>
> @ 0x18e2181 impala::ThriftThread::RunRunnable()
>
> @ 0x18e3885 boost::_mfi::mf2<>::operator()()
>
> @ 0x18e371b boost::_bi::list3<>::operator()<>()
>
> @ 0x18e3467 boost::_bi::bind_t<>::operator()()
>
> @ 0x18e337a boost::detail::function::void_
> function_obj_invoker0<>::invoke()
>
> @ 0x192761c boost::function0<>::operator()()
>
> @ 0x1c3ebf7 impala::Thread::SuperviseThread()
>
> @ 0x1c470cd boost::_bi::list5<>::operator()<>()
>
> @ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
>
> @ 0x1c46fb4 boost::detail::thread_data<>::run()
>
> @ 0x2eedb4a thread_proxy
>
> @ 0x7fda1dbb16ba start_thread
>
> @ 0x7fda1d8e741d clone
>
> Wrote minidump to /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-
> 47b42022.dmp
>
>
>
> *impalad.FATAL*
>
>
>
> Log file created at: 2018/05/07 09:46:12
>
> Running on machine: slave2
>
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>
> F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
> log_entry.count > 0 (-1831809966 vs. 0)
>
>
>
> *Impalad.INFO*
>
> edentials={real_user=root}} blocked reactor thread for 34288.6us
>
> I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> (slave5), user_credentials={real_user=root}} blocked reactor thread for
> 35859.8us
>
> I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> (slave5), user_credentials={real_user=root}} blocked reactor thread for
> 40664.9us
>
> I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC
> call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050
> (slave5), user_credentials={real_user=root}} blocked reactor thread for
> 49514.6us
>
> I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission control
> resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
>
> F0507 09:46:12.673912 29258 error-util.cc:148] Check failed:
> log_entry.count > 0 (-1831809966 vs. 0)
>
> Wrote minidump to /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-
> 47b42022.dmp
>
>
>
> *Note*:
>
> We are executing the queries on 8 node cluster with the following
> configuration
>
> Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
> Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
>
>
>
>
>
> --
>
> Regards,
>
> Geetika Gupta
>
--
Regards,
Geetika Gupta
答复: Issue in data loading in Impala + Kudu
Posted by helifu <hz...@corp.netease.com>.
Hi Geetika,
It would be better to ask this question in impala user mail list. Here is the impala community: https://impala.apache.org/community.html
何李夫
2018-05-07 13:56:02
发件人: user-return-1353-hzhelifu=corp.netease.com@kudu.apache.org <us...@kudu.apache.org> 代表 Geetika Gupta
发送时间: 2018年5月7日 13:42
收件人: user@kudu.apache.org
主题: Issue in data loading in Impala + Kudu
Hi community,
I was trying to load 500GB of TPCH data into kudu table using the following query:
insert into lineitem select * from PARQUETIMPALA500.LINEITEM
While executing the query for around 17 hrs it got cancelled as the impalad process of that machine got aborted. Here are the logs of the impalad process.
impalad.ERROR
Log file created at: 2018/05/06 13:40:34
Running on machine: slave2
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0506 13:40:34.097759 28730 logging.cc:121] stderr will be logged to this file.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/softwares/impala/fe/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/softwares/impala/testdata/target/dependency/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/05/06 13:40:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/05/06 13:40:36 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
tcmalloc: large alloc 1073741824 bytes == 0x484434000 @ 0x4135176 0x7fd9e9fc3929
tcmalloc: large alloc 2147483648 bytes == 0x7fd540f18000 @ 0x4135176 0x7fd9e9fc3929
F0507 09:46:12.673912 29258 error-util.cc:148] Check failed: log_entry.count > 0 (-1831809966 vs. 0)
*** Check failure stack trace: ***
@ 0x3fc0c0d google::LogMessage::Fail()
@ 0x3fc24b2 google::LogMessage::SendToLog()
@ 0x3fc05e7 google::LogMessage::Flush()
@ 0x3fc3bae google::LogMessageFatal::~LogMessageFatal()
@ 0x1bbcb31 impala::PrintErrorMap()
@ 0x1bbcd07 impala::PrintErrorMapToString()
@ 0x2decbd7 impala::Coordinator::GetErrorLog()
@ 0x1a8d634 impala::ImpalaServer::UnregisterQuery()
@ 0x1b29264 impala::ImpalaServer::CloseOperation()
@ 0x2c5ce86 apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseOperation()
@ 0x2c56b8c apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
@ 0x2c2fcb1 impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
@ 0x16fdb20 apache::thrift::TDispatchProcessor::process()
@ 0x18ea6b3 apache::thrift::server::TAcceptQueueServer::Task::run()
@ 0x18e2181 impala::ThriftThread::RunRunnable()
@ 0x18e3885 boost::_mfi::mf2<>::operator()()
@ 0x18e371b boost::_bi::list3<>::operator()<>()
@ 0x18e3467 boost::_bi::bind_t<>::operator()()
@ 0x18e337a boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x192761c boost::function0<>::operator()()
@ 0x1c3ebf7 impala::Thread::SuperviseThread()
@ 0x1c470cd boost::_bi::list5<>::operator()<>()
@ 0x1c46ff1 boost::_bi::bind_t<>::operator()()
@ 0x1c46fb4 boost::detail::thread_data<>::run()
@ 0x2eedb4a thread_proxy
@ 0x7fda1dbb16ba start_thread
@ 0x7fda1d8e741d clone
Wrote minidump to /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
impalad.FATAL
Log file created at: 2018/05/07 09:46:12
Running on machine: slave2
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0507 09:46:12.673912 29258 error-util.cc:148] Check failed: log_entry.count > 0 (-1831809966 vs. 0)
Impalad.INFO
edentials={real_user=root}} blocked reactor thread for 34288.6us
I0507 09:38:14.943245 29882 outbound_call.cc:288] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050 <http://136.243.74.42:7050> (slave5), user_credentials={real_user=root}} blocked reactor thread for 35859.8us
I0507 09:38:15.942150 29882 outbound_call.cc:288] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050 <http://136.243.74.42:7050> (slave5), user_credentials={real_user=root}} blocked reactor thread for 40664.9us
I0507 09:38:17.495046 29882 outbound_call.cc:288] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=136.243.74.42:7050 <http://136.243.74.42:7050> (slave5), user_credentials={real_user=root}} blocked reactor thread for 49514.6us
I0507 09:46:12.664149 4507 coordinator.cc:783] Release admission control resources for query_id=3e4a4c646800e1d9:c859bb7f00000000
F0507 09:46:12.673912 29258 error-util.cc:148] Check failed: log_entry.count > 0 (-1831809966 vs. 0)
Wrote minidump to /tmp/minidumps/impalad/a9113d9b-bc3d-488a-1feebf9b-47b42022.dmp
Note:
We are executing the queries on 8 node cluster with the following configuration
Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each,
Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
--
Regards,
Geetika Gupta