You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "xubo245 (JIRA)" <ji...@apache.org> on 2018/04/23 08:36:00 UTC
[jira] [Updated] (CARBONDATA-2382) OutOfMemoryError when use search
mode in ConcurrentQueryBenchmark
[ https://issues.apache.org/jira/browse/CARBONDATA-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
xubo245 updated CARBONDATA-2382:
--------------------------------
Description:
{code:java}
18/04/23 16:24:49 WARN Utils: Your hostname, ecs-909c resolves to a loopback address: 127.0.0.1; using 192.168.0.206 instead (on interface eth0)
18/04/23 16:24:49 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
18/04/23 16:24:50 INFO SparkContext: Submitted application: totalNum: 10000000 threadNum: 16 taskNum: 100 resultIsEmpty: true file path: /tmp/carbondata runInLocal: true generateFile: true deleteFile: false
18/04/23 16:24:50 INFO SecurityManager: Changing view acls to: root
18/04/23 16:24:50 INFO SecurityManager: Changing modify acls to: root
18/04/23 16:24:50 INFO SecurityManager: Changing view acls groups to:
18/04/23 16:24:50 INFO SecurityManager: Changing modify acls groups to:
18/04/23 16:24:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
18/04/23 16:24:50 INFO Utils: Successfully started service 'sparkDriver' on port 35444.
18/04/23 16:24:50 INFO SparkEnv: Registering MapOutputTracker
18/04/23 16:24:50 INFO SparkEnv: Registering BlockManagerMaster
18/04/23 16:24:50 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/04/23 16:24:50 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/04/23 16:24:50 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-186ff215-cd17-4b23-9fe0-c55e169f5d87
18/04/23 16:24:50 INFO MemoryStore: MemoryStore started with capacity 1957.8 MB
18/04/23 16:24:50 INFO SparkEnv: Registering OutputCommitCoordinator
18/04/23 16:24:50 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/04/23 16:24:50 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.206:4040
18/04/23 16:24:50 INFO Executor: Starting executor ID driver on host localhost
18/04/23 16:24:50 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46504.
18/04/23 16:24:50 INFO NettyBlockTransferService: Server created on 192.168.0.206:46504
18/04/23 16:24:50 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/04/23 16:24:50 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.206, 46504, None)
18/04/23 16:24:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.206:46504 with 1957.8 MB RAM, BlockManagerId(driver, 192.168.0.206, 46504, None)
18/04/23 16:24:50 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.206, 46504, None)
18/04/23 16:24:50 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.206, 46504, None)
18/04/23 16:24:51 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/huawei/xubo/git/carbondata1/spark-warehouse').
18/04/23 16:24:51 INFO SharedState: Warehouse path is 'file:/huawei/xubo/git/carbondata1/spark-warehouse'.
18/04/23 16:24:51 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
18/04/23 16:24:52 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/04/23 16:24:52 INFO ObjectStore: ObjectStore, initialize called
18/04/23 16:24:52 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/04/23 16:24:52 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
18/04/23 16:24:53 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/04/23 16:24:54 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/04/23 16:24:54 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/04/23 16:24:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/04/23 16:24:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/04/23 16:24:55 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
18/04/23 16:24:55 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/04/23 16:24:55 INFO ObjectStore: Initialized ObjectStore
18/04/23 16:24:55 INFO HiveMetaStore: Added admin role in metastore
18/04/23 16:24:55 INFO HiveMetaStore: Added public role in metastore
18/04/23 16:24:55 INFO HiveMetaStore: No user is added in admin role, since config is empty
18/04/23 16:24:55 INFO HiveMetaStore: 0: get_all_databases
18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_all_databases
18/04/23 16:24:55 INFO HiveMetaStore: 0: get_functions: db=default pat=*
18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
18/04/23 16:24:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
18/04/23 16:24:55 INFO SessionState: Created local directory: /tmp/a2a6b2d9-89b8-49ed-a1e4-510d33e89963_resources
18/04/23 16:24:55 INFO SessionState: Created HDFS directory: /tmp/hive/root/a2a6b2d9-89b8-49ed-a1e4-510d33e89963
18/04/23 16:24:55 INFO SessionState: Created local directory: /tmp/root/a2a6b2d9-89b8-49ed-a1e4-510d33e89963
18/04/23 16:24:55 INFO SessionState: Created HDFS directory: /tmp/hive/root/a2a6b2d9-89b8-49ed-a1e4-510d33e89963/_tmp_space.db
18/04/23 16:24:55 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/huawei/xubo/git/carbondata1/spark-warehouse
18/04/23 16:24:55 INFO HiveMetaStore: 0: get_database: default
18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_database: default
18/04/23 16:24:55 INFO HiveMetaStore: 0: get_database: global_temp
18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_database: global_temp
18/04/23 16:24:55 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
18/04/23 16:24:56 INFO SessionState: Created local directory: /tmp/fe452d60-a43e-4edf-8945-600c341bb02c_resources
18/04/23 16:24:56 INFO SessionState: Created HDFS directory: /tmp/hive/root/fe452d60-a43e-4edf-8945-600c341bb02c
18/04/23 16:24:56 INFO SessionState: Created local directory: /tmp/root/fe452d60-a43e-4edf-8945-600c341bb02c
18/04/23 16:24:56 INFO SessionState: Created HDFS directory: /tmp/hive/root/fe452d60-a43e-4edf-8945-600c341bb02c/_tmp_space.db
18/04/23 16:24:56 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/huawei/xubo/git/carbondata1/spark-warehouse
18/04/23 16:24:56 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
Environment information:
spark.master: local[8]
spark.driver.cores: default value
spark.driver.memory: default value
spark.executor.cores: default value
spark.executor.memory: default value
spark.executor.instances: default value
SPARK_VERSION:2.2.1
CARBONDATA_VERSION:1.4.0-SNAPSHOT
Parameters information:
totalNum: 10000000 threadNum: 16 taskNum: 100 resultIsEmpty: true file path: /tmp/carbondata runInLocal: true generateFile: true deleteFile: false
Start generate 10000000 records, schema: StructType(StructField(id,StringType,false), StructField(city,StringType,false), StructField(country,StringType,false), StructField(planet,StringType,false), StructField(m1,ShortType,false), StructField(m2,IntegerType,false), StructField(m3,LongType,false), StructField(m4,DoubleType,false), StructField(m5,DecimalType(30,10),false))
Num10000000_comparetest_parquet completed, time: 33.757 sec
18/04/23 16:25:37 AUDIT CarbonCreateTableCommand: [ecs-909c][root][Thread-1]Creating Table with Database name [default] and Table name [num10000000_comparetest_carbonv3]
18/04/23 16:25:38 AUDIT CarbonCreateTableCommand: [ecs-909c][root][Thread-1]Table created with Database name [default] and Table name [num10000000_comparetest_carbonv3]
18/04/23 16:25:38 AUDIT CarbonDataRDDFactory$: [ecs-909c][root][Thread-1]Data load request has been received for table default.num10000000_comparetest_carbonv3
18/04/23 16:26:36 AUDIT CarbonDataRDDFactory$: [ecs-909c][root][Thread-1]Data load is successful for default.num10000000_comparetest_carbonv3
Num10000000_comparetest_carbonV3 completed, time: 59.577 sec
Start running queries for Num10000000_comparetest_carbonV3...
Min: min time Max: max time 90%: 90% time 99%: 99% time Avg: average time Count: number of result Query X: running different query sql Result: show it when ResultIsEmpty is false Total execute time: total runtime
18/04/23 16:27:12 ERROR Inbox: Ignoring error
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:75)
at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.<init>(ChunkRowIterator.java:40)
at org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:84)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleRequest(SearchRequestHandler.java:95)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleSearch(SearchRequestHandler.java:62)
at org.apache.spark.search.Searcher$$anonfun$receiveAndReply$1.applyOrElse(Searcher.scala:41)
at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:105)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:216)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:71)
... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.carbondata.core.datastore.impl.FileReaderImpl.readByteBuffer(FileReaderImpl.java:200)
at org.apache.carbondata.core.datastore.chunk.reader.measure.v3.CompressedMeasureChunkFileBasedReaderV3.readRawMeasureChunksInGroup(CompressedMeasureChunkFileBasedReaderV3.java:157)
at org.apache.carbondata.core.datastore.chunk.reader.measure.AbstractMeasureChunkReaderV2V3Format.readRawMeasureChunks(AbstractMeasureChunkReaderV2V3Format.java:77)
at org.apache.carbondata.core.indexstore.blockletindex.BlockletDataRefNode.readMeasureChunks(BlockletDataRefNode.java:147)
at org.apache.carbondata.core.scan.scanner.impl.BlockletFilterScanner.executeFilter(BlockletFilterScanner.java:278)
at org.apache.carbondata.core.scan.scanner.impl.BlockletFilterScanner.scanBlocklet(BlockletFilterScanner.java:101)
at org.apache.carbondata.core.scan.processor.BlockScan.scan(BlockScan.java:69)
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator$1.call(AbstractSearchModeResultIterator.java:59)
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator$1.call(AbstractSearchModeResultIterator.java:53)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
18/04/23 16:27:12 ERROR Inbox: Ignoring error
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:75)
at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.<init>(ChunkRowIterator.java:40)
at org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:84)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleRequest(SearchRequestHandler.java:95)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleSearch(SearchRequestHandler.java:62)
at org.apache.spark.search.Searcher$$anonfun$receiveAndReply$1.applyOrElse(Searcher.scala:41)
at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:105)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:216)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:71)
... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
18/04/23 16:27:27 ERROR Inbox: Ignoring error
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:75)
at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.<init>(ChunkRowIterator.java:40)
at org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:84)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleRequest(SearchRequestHandler.java:95)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleSearch(SearchRequestHandler.java:62)
at org.apache.spark.search.Searcher$$anonfun$receiveAndReply$1.applyOrElse(Searcher.scala:41)
at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:105)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:216)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:71)
... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
18/04/23 16:27:27 ERROR CarbonSession: Exception when executing search mode: Exception thrown in awaitResult: , fallback to SparkSQL
18/04/23 16:27:27 ERROR CarbonSession: Exception when executing search mode: Exception thrown in awaitResult: , fallback to SparkSQL
18/04/23 16:27:28 ERROR CarbonSession: Exception when executing search mode: Exception thrown in awaitResult: , fallback to SparkSQL
{code}
was:
{code:java}
/huawei/tools/jdk1.8.0_161/bin/java -javaagent:/huawei/tools/idea-IC-171.4424.56/lib/idea_rt.jar=35309:/huawei/tools/idea-IC-171.4424.56/bin -Dfile.encoding=UTF-8 -classpath /huawei/tools/jdk1.8.0_161/jre/lib/charsets.jar:/huawei/tools/jdk1.8.0_161/jre/lib/deploy.jar:/huawei/tools/jdk1.8.0_161/jre/lib/ext/cldrdata.jar:/huawei/tools/jdk1.8.0_161/jre/lib/ext/dnsns.jar:/huawei/tools/jdk1.8.0_161/jre/lib/ext/jaccess.jar:/huawei/tools/jdk1.8.0_161/jre/lib/ext/jfxrt.jar:/huawei/tools/jdk1.8.0_161/jre/lib/ext/localedata.jar:/huawei/tools/jdk1.8.0_161/jre/lib/ext/nashorn.jar:/huawei/tools/jdk1.8.0_161/jre/lib/ext/sunec.jar:/huawei/tools/jdk1.8.0_161/jre/lib/ext/sunjce_provider.jar:/huawei/tools/jdk1.8.0_161/jre/lib/ext/sunpkcs11.jar:/huawei/tools/jdk1.8.0_161/jre/lib/ext/zipfs.jar:/huawei/tools/jdk1.8.0_161/jre/lib/javaws.jar:/huawei/tools/jdk1.8.0_161/jre/lib/jce.jar:/huawei/tools/jdk1.8.0_161/jre/lib/jfr.jar:/huawei/tools/jdk1.8.0_161/jre/lib/jfxswt.jar:/huawei/tools/jdk1.8.0_161/jre/lib/jsse.jar:/huawei/tools/jdk1.8.0_161/jre/lib/management-agent.jar:/huawei/tools/jdk1.8.0_161/jre/lib/plugin.jar:/huawei/tools/jdk1.8.0_161/jre/lib/resources.jar:/huawei/tools/jdk1.8.0_161/jre/lib/rt.jar:/huawei/xubo/git/carbondata1/examples/spark2/target/classes:/huawei/xubo/git/carbondata1/integration/spark2/target/classes:/huawei/xubo/git/carbondata1/integration/spark-common/target/classes:/huawei/xubo/git/carbondata1/store/sdk/target/classes:/huawei/xubo/git/carbondata1/store/search/target/classes:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-aws/2.7.2/hadoop-aws-2.7.2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-common/2.7.2/hadoop-common-2.7.2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-annotations/2.7.2/hadoop-annotations-2.7.2.jar:/huawei/tools/jdk1.8.0_161/lib/tools.jar:/huawei/tools/mavenRepo/carbondata1/com/google/guava/guava/11.0.2/guava-11.0.2.jar:/huawei/tools/mavenRepo/carbondata1/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/huawei/tools/mavenRepo/carbondata1/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/huawei/tools/mavenRepo/carbondata1/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/huawei/tools/mavenRepo/carbondata1/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/huawei/tools/mavenRepo/carbondata1/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/huawei/tools/mavenRepo/carbondata1/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar:/huawei/tools/mavenRepo/carbondata1/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/huawei/tools/mavenRepo/carbondata1/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/huawei/tools/mavenRepo/carbondata1/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/huawei/tools/mavenRepo/carbondata1/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/huawei/tools/mavenRepo/carbondata1/javax/activation/activation/1.1/activation-1.1.jar:/huawei/tools/mavenRepo/carbondata1/org/codehaus/jackson/jackson-jaxrs/1.8.3/jackson-jaxrs-1.8.3.jar:/huawei/tools/mavenRepo/carbondata1/org/codehaus/jackson/jackson-xc/1.8.3/jackson-xc-1.8.3.jar:/huawei/tools/mavenRepo/carbondata1/com/sun/jersey/jersey-server/1.9/jersey-server-1.9.jar:/huawei/tools/mavenRepo/carbondata1/asm/asm/3.1/asm-3.1.jar:/huawei/tools/mavenRepo/carbondata1/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/huawei/tools/mavenRepo/carbondata1/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/huawei/tools/mavenRepo/carbondata1/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/huawei/tools/mavenRepo/carbondata1/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/huawei/tools/mavenRepo/carbondata1/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/huawei/tools/mavenRepo/carbondata1/com/google/code/gson/gson/2.2.4/gson-2.2.4.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-auth/2.7.2/hadoop-auth-2.7.2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/directory/server/apacheds-kerberos-codec/2.0.0-M15/apacheds-kerberos-codec-2.0.0-M15.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/directory/server/apacheds-i18n/2.0.0-M15/apacheds-i18n-2.0.0-M15.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/directory/api/api-asn1-api/1.0.0-M20/api-asn1-api-1.0.0-M20.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/directory/api/api-util/1.0.0-M20/api-util-1.0.0-M20.jar:/huawei/tools/mavenRepo/carbondata1/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/curator/curator-client/2.7.1/curator-client-2.7.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/htrace/htrace-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar:/huawei/tools/mavenRepo/carbondata1/org/tukaani/xz/1.0/xz-1.0.jar:/huawei/tools/mavenRepo/carbondata1/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar:/huawei/tools/mavenRepo/carbondata1/joda-time/joda-time/2.9.9/joda-time-2.9.9.jar:/huawei/tools/mavenRepo/carbondata1/net/java/dev/jets3t/jets3t/0.9.0/jets3t-0.9.0.jar:/huawei/tools/mavenRepo/carbondata1/com/jamesmurty/utils/java-xmlbuilder/0.4/java-xmlbuilder-0.4.jar:/huawei/xubo/git/carbondata1/streaming/target/classes:/huawei/xubo/git/carbondata1/hadoop/target/classes:/huawei/xubo/git/carbondata1/processing/target/classes:/huawei/xubo/git/carbondata1/core/target/classes:/huawei/tools/mavenRepo/carbondata1/org/apache/carbondata/carbondata-format/1.4.0-SNAPSHOT/carbondata-format-1.4.0-20180209.090815-4.jar:/huawei/xubo/git/carbondata1/common/target/classes:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-hdfs/2.7.2/hadoop-hdfs-2.7.2.jar:/huawei/tools/mavenRepo/carbondata1/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar:/huawei/tools/mavenRepo/carbondata1/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar:/huawei/tools/mavenRepo/carbondata1/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-sql_2.11/2.2.1/spark-sql_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/com/univocity/univocity-parsers/2.2.1/univocity-parsers-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-sketch_2.11/2.2.1/spark-sketch_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-core_2.11/2.2.1/spark-core_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/avro/avro/1.7.7/avro-1.7.7.jar:/huawei/tools/mavenRepo/carbondata1/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/avro/avro-mapred/1.7.7/avro-mapred-1.7.7-hadoop2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7-tests.jar:/huawei/tools/mavenRepo/carbondata1/com/twitter/chill_2.11/0.8.0/chill_2.11-0.8.0.jar:/huawei/tools/mavenRepo/carbondata1/com/esotericsoftware/kryo-shaded/3.0.3/kryo-shaded-3.0.3.jar:/huawei/tools/mavenRepo/carbondata1/com/esotericsoftware/minlog/1.3.0/minlog-1.3.0.jar:/huawei/tools/mavenRepo/carbondata1/org/objenesis/objenesis/2.1/objenesis-2.1.jar:/huawei/tools/mavenRepo/carbondata1/com/twitter/chill-java/0.8.0/chill-java-0.8.0.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-client/2.6.5/hadoop-client-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-mapreduce-client-app/2.6.5/hadoop-mapreduce-client-app-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-mapreduce-client-common/2.6.5/hadoop-mapreduce-client-common-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-yarn-client/2.6.5/hadoop-yarn-client-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-yarn-server-common/2.6.5/hadoop-yarn-server-common-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.6.5/hadoop-mapreduce-client-shuffle-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-yarn-api/2.6.5/hadoop-yarn-api-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-mapreduce-client-core/2.7.2/hadoop-mapreduce-client-core-2.7.2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-yarn-common/2.7.2/hadoop-yarn-common-2.7.2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.6.5/hadoop-mapreduce-client-jobclient-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-launcher_2.11/2.2.1/spark-launcher_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-network-common_2.11/2.2.1/spark-network-common_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-network-shuffle_2.11/2.2.1/spark-network-shuffle_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-unsafe_2.11/2.2.1/spark-unsafe_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar:/huawei/tools/mavenRepo/carbondata1/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/commons/commons-math3/3.4.1/commons-math3-3.4.1.jar:/huawei/tools/mavenRepo/carbondata1/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/huawei/tools/mavenRepo/carbondata1/org/slf4j/slf4j-api/1.7.16/slf4j-api-1.7.16.jar:/huawei/tools/mavenRepo/carbondata1/org/slf4j/jcl-over-slf4j/1.7.16/jcl-over-slf4j-1.7.16.jar:/huawei/tools/mavenRepo/carbondata1/log4j/log4j/1.2.17/log4j-1.2.17.jar:/huawei/tools/mavenRepo/carbondata1/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar:/huawei/tools/mavenRepo/carbondata1/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/huawei/tools/mavenRepo/carbondata1/org/xerial/snappy/snappy-java/1.1.2.6/snappy-java-1.1.2.6.jar:/huawei/tools/mavenRepo/carbondata1/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar:/huawei/tools/mavenRepo/carbondata1/org/roaringbitmap/RoaringBitmap/0.5.11/RoaringBitmap-0.5.11.jar:/huawei/tools/mavenRepo/carbondata1/commons-net/commons-net/2.2/commons-net-2.2.jar:/huawei/tools/mavenRepo/carbondata1/org/json4s/json4s-jackson_2.11/3.2.11/json4s-jackson_2.11-3.2.11.jar:/huawei/tools/mavenRepo/carbondata1/org/json4s/json4s-core_2.11/3.2.11/json4s-core_2.11-3.2.11.jar:/huawei/tools/mavenRepo/carbondata1/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar:/huawei/tools/mavenRepo/carbondata1/org/scala-lang/scalap/2.11.8/scalap-2.11.8.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/jersey/core/jersey-client/2.22.2/jersey-client-2.22.2.jar:/huawei/tools/mavenRepo/carbondata1/javax/ws/rs/javax.ws.rs-api/2.0.1/javax.ws.rs-api-2.0.1.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/hk2/hk2-api/2.4.0-b34/hk2-api-2.4.0-b34.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/hk2/hk2-utils/2.4.0-b34/hk2-utils-2.4.0-b34.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/hk2/external/aopalliance-repackaged/2.4.0-b34/aopalliance-repackaged-2.4.0-b34.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/hk2/external/javax.inject/2.4.0-b34/javax.inject-2.4.0-b34.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/hk2/hk2-locator/2.4.0-b34/hk2-locator-2.4.0-b34.jar:/huawei/tools/mavenRepo/carbondata1/org/javassist/javassist/3.18.1-GA/javassist-3.18.1-GA.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/jersey/core/jersey-common/2.22.2/jersey-common-2.22.2.jar:/huawei/tools/mavenRepo/carbondata1/javax/annotation/javax.annotation-api/1.2/javax.annotation-api-1.2.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/jersey/bundles/repackaged/jersey-guava/2.22.2/jersey-guava-2.22.2.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/hk2/osgi-resource-locator/1.0.1/osgi-resource-locator-1.0.1.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/jersey/core/jersey-server/2.22.2/jersey-server-2.22.2.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/jersey/media/jersey-media-jaxb/2.22.2/jersey-media-jaxb-2.22.2.jar:/huawei/tools/mavenRepo/carbondata1/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/jersey/containers/jersey-container-servlet/2.22.2/jersey-container-servlet-2.22.2.jar:/huawei/tools/mavenRepo/carbondata1/org/glassfish/jersey/containers/jersey-container-servlet-core/2.22.2/jersey-container-servlet-core-2.22.2.jar:/huawei/tools/mavenRepo/carbondata1/io/netty/netty-all/4.0.43.Final/netty-all-4.0.43.Final.jar:/huawei/tools/mavenRepo/carbondata1/io/netty/netty/3.9.9.Final/netty-3.9.9.Final.jar:/huawei/tools/mavenRepo/carbondata1/com/clearspring/analytics/stream/2.7.0/stream-2.7.0.jar:/huawei/tools/mavenRepo/carbondata1/io/dropwizard/metrics/metrics-core/3.1.2/metrics-core-3.1.2.jar:/huawei/tools/mavenRepo/carbondata1/io/dropwizard/metrics/metrics-jvm/3.1.2/metrics-jvm-3.1.2.jar:/huawei/tools/mavenRepo/carbondata1/io/dropwizard/metrics/metrics-json/3.1.2/metrics-json-3.1.2.jar:/huawei/tools/mavenRepo/carbondata1/io/dropwizard/metrics/metrics-graphite/3.1.2/metrics-graphite-3.1.2.jar:/huawei/tools/mavenRepo/carbondata1/com/fasterxml/jackson/module/jackson-module-scala_2.11/2.6.5/jackson-module-scala_2.11-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/com/fasterxml/jackson/module/jackson-module-paranamer/2.6.5/jackson-module-paranamer-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/huawei/tools/mavenRepo/carbondata1/oro/oro/2.0.8/oro-2.0.8.jar:/huawei/tools/mavenRepo/carbondata1/net/razorvine/pyrolite/4.13/pyrolite-4.13.jar:/huawei/tools/mavenRepo/carbondata1/net/sf/py4j/py4j/0.10.4/py4j-0.10.4.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/commons/commons-crypto/1.0.0/commons-crypto-1.0.0.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-catalyst_2.11/2.2.1/spark-catalyst_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/codehaus/janino/janino/3.0.0/janino-3.0.0.jar:/huawei/tools/mavenRepo/carbondata1/org/codehaus/janino/commons-compiler/3.0.0/commons-compiler-3.0.0.jar:/huawei/tools/mavenRepo/carbondata1/org/antlr/antlr4-runtime/4.5.3/antlr4-runtime-4.5.3.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-tags_2.11/2.2.1/spark-tags_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/parquet/parquet-column/1.8.2/parquet-column-1.8.2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/parquet/parquet-common/1.8.2/parquet-common-1.8.2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/parquet/parquet-encoding/1.8.2/parquet-encoding-1.8.2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/parquet/parquet-hadoop/1.8.2/parquet-hadoop-1.8.2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/parquet/parquet-format/2.3.1/parquet-format-2.3.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/parquet/parquet-jackson/1.8.2/parquet-jackson-1.8.2.jar:/huawei/tools/mavenRepo/carbondata1/org/codehaus/jackson/jackson-mapper-asl/1.9.11/jackson-mapper-asl-1.9.11.jar:/huawei/tools/mavenRepo/carbondata1/org/codehaus/jackson/jackson-core-asl/1.9.11/jackson-core-asl-1.9.11.jar:/huawei/tools/mavenRepo/carbondata1/com/fasterxml/jackson/core/jackson-databind/2.6.5/jackson-databind-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar:/huawei/tools/mavenRepo/carbondata1/com/fasterxml/jackson/core/jackson-core/2.6.5/jackson-core-2.6.5.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/xbean/xbean-asm5-shaded/4.4/xbean-asm5-shaded-4.4.jar:/huawei/tools/mavenRepo/carbondata1/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-hive-thriftserver_2.11/2.2.1/spark-hive-thriftserver_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-hive_2.11/2.2.1/spark-hive_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/com/twitter/parquet-hadoop-bundle/1.6.0/parquet-hadoop-bundle-1.6.0.jar:/huawei/tools/mavenRepo/carbondata1/org/spark-project/hive/hive-exec/1.2.1.spark2/hive-exec-1.2.1.spark2.jar:/huawei/tools/mavenRepo/carbondata1/javolution/javolution/5.5.1/javolution-5.5.1.jar:/huawei/tools/mavenRepo/carbondata1/log4j/apache-log4j-extras/1.2.17/apache-log4j-extras-1.2.17.jar:/huawei/tools/mavenRepo/carbondata1/org/antlr/antlr-runtime/3.4/antlr-runtime-3.4.jar:/huawei/tools/mavenRepo/carbondata1/org/antlr/stringtemplate/3.2.1/stringtemplate-3.2.1.jar:/huawei/tools/mavenRepo/carbondata1/antlr/antlr/2.7.7/antlr-2.7.7.jar:/huawei/tools/mavenRepo/carbondata1/org/antlr/ST4/4.0.4/ST4-4.0.4.jar:/huawei/tools/mavenRepo/carbondata1/com/googlecode/javaewah/JavaEWAH/0.3.2/JavaEWAH-0.3.2.jar:/huawei/tools/mavenRepo/carbondata1/org/iq80/snappy/snappy/0.2/snappy-0.2.jar:/huawei/tools/mavenRepo/carbondata1/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/huawei/tools/mavenRepo/carbondata1/net/sf/opencsv/opencsv/2.3/opencsv-2.3.jar:/huawei/tools/mavenRepo/carbondata1/org/spark-project/hive/hive-metastore/1.2.1.spark2/hive-metastore-1.2.1.spark2.jar:/huawei/tools/mavenRepo/carbondata1/com/jolbox/bonecp/0.8.0.RELEASE/bonecp-0.8.0.RELEASE.jar:/huawei/tools/mavenRepo/carbondata1/org/datanucleus/datanucleus-api-jdo/3.2.6/datanucleus-api-jdo-3.2.6.jar:/huawei/tools/mavenRepo/carbondata1/org/datanucleus/datanucleus-rdbms/3.2.9/datanucleus-rdbms-3.2.9.jar:/huawei/tools/mavenRepo/carbondata1/commons-pool/commons-pool/1.5.4/commons-pool-1.5.4.jar:/huawei/tools/mavenRepo/carbondata1/commons-dbcp/commons-dbcp/1.4/commons-dbcp-1.4.jar:/huawei/tools/mavenRepo/carbondata1/javax/jdo/jdo-api/3.0.1/jdo-api-3.0.1.jar:/huawei/tools/mavenRepo/carbondata1/javax/transaction/jta/1.1/jta-1.1.jar:/huawei/tools/mavenRepo/carbondata1/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/calcite/calcite-avatica/1.2.0-incubating/calcite-avatica-1.2.0-incubating.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/calcite/calcite-core/1.2.0-incubating/calcite-core-1.2.0-incubating.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/calcite/calcite-linq4j/1.2.0-incubating/calcite-linq4j-1.2.0-incubating.jar:/huawei/tools/mavenRepo/carbondata1/net/hydromatic/eigenbase-properties/1.1.5/eigenbase-properties-1.1.5.jar:/huawei/tools/mavenRepo/carbondata1/org/jodd/jodd-core/3.5.2/jodd-core-3.5.2.jar:/huawei/tools/mavenRepo/carbondata1/org/datanucleus/datanucleus-core/3.2.10/datanucleus-core-3.2.10.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/thrift/libthrift/0.9.3/libthrift-0.9.3.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/derby/derby/10.12.1.1/derby-10.12.1.1.jar:/huawei/tools/mavenRepo/carbondata1/org/spark-project/hive/hive-cli/1.2.1.spark2/hive-cli-1.2.1.spark2.jar:/huawei/tools/mavenRepo/carbondata1/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/huawei/tools/mavenRepo/carbondata1/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/huawei/tools/mavenRepo/carbondata1/org/spark-project/hive/hive-jdbc/1.2.1.spark2/hive-jdbc-1.2.1.spark2.jar:/huawei/tools/mavenRepo/carbondata1/org/spark-project/hive/hive-beeline/1.2.1.spark2/hive-beeline-1.2.1.spark2.jar:/huawei/tools/mavenRepo/carbondata1/commons-io/commons-io/2.4/commons-io-2.4.jar:/huawei/tools/mavenRepo/carbondata1/net/sf/supercsv/super-csv/2.2.0/super-csv-2.2.0.jar:/huawei/tools/mavenRepo/carbondata1/net/sf/jpam/jpam/1.1/jpam-1.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-repl_2.11/2.2.1/spark-repl_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-mllib_2.11/2.2.1/spark-mllib_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-graphx_2.11/2.2.1/spark-graphx_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/com/github/fommil/netlib/core/1.1.2/core-1.1.2.jar:/huawei/tools/mavenRepo/carbondata1/net/sourceforge/f2j/arpack_combined_all/0.1/arpack_combined_all-0.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-mllib-local_2.11/2.2.1/spark-mllib-local_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/scalanlp/breeze_2.11/0.13.2/breeze_2.11-0.13.2.jar:/huawei/tools/mavenRepo/carbondata1/org/scalanlp/breeze-macros_2.11/0.13.2/breeze-macros_2.11-0.13.2.jar:/huawei/tools/mavenRepo/carbondata1/com/github/rwl/jtransforms/2.4.0/jtransforms-2.4.0.jar:/huawei/tools/mavenRepo/carbondata1/org/spire-math/spire_2.11/0.13.0/spire_2.11-0.13.0.jar:/huawei/tools/mavenRepo/carbondata1/org/spire-math/spire-macros_2.11/0.13.0/spire-macros_2.11-0.13.0.jar:/huawei/tools/mavenRepo/carbondata1/org/typelevel/machinist_2.11/0.6.1/machinist_2.11-0.6.1.jar:/huawei/tools/mavenRepo/carbondata1/com/chuusai/shapeless_2.11/2.3.2/shapeless_2.11-2.3.2.jar:/huawei/tools/mavenRepo/carbondata1/org/typelevel/macro-compat_2.11/1.1.1/macro-compat_2.11-1.1.1.jar:/huawei/tools/mavenRepo/carbondata1/org/jpmml/pmml-model/1.2.15/pmml-model-1.2.15.jar:/huawei/tools/mavenRepo/carbondata1/org/jpmml/pmml-schema/1.2.15/pmml-schema-1.2.15.jar:/huawei/tools/mavenRepo/carbondata1/org/scala-lang/scala-compiler/2.11.8/scala-compiler-2.11.8.jar:/huawei/tools/mavenRepo/carbondata1/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.jar:/huawei/tools/mavenRepo/carbondata1/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar:/huawei/tools/mavenRepo/carbondata1/jline/jline/2.12.1/jline-2.12.1.jar:/huawei/tools/mavenRepo/carbondata1/org/slf4j/jul-to-slf4j/1.7.16/jul-to-slf4j-1.7.16.jar:/huawei/tools/mavenRepo/carbondata1/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/spark/spark-streaming_2.11/2.2.1/spark-streaming_2.11-2.2.1.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/httpcomponents/httpclient/4.2/httpclient-4.2.jar:/huawei/tools/mavenRepo/carbondata1/org/apache/httpcomponents/httpcore/4.2/httpcore-4.2.jar:/huawei/tools/mavenRepo/carbondata1/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/huawei/tools/mavenRepo/carbondata1/commons-codec/commons-codec/1.6/commons-codec-1.6.jar:/huawei/tools/mavenRepo/carbondata1/org/scala-lang/modules/scala-xml_2.11/1.0.2/scala-xml_2.11-1.0.2.jar org.apache.carbondata.benchmark.ConcurrentQueryBenchmark
log4j:WARN No appenders could be found for logger (org.apache.carbondata.core.util.CarbonProperties).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/04/23 16:24:49 INFO SparkContext: Running Spark version 2.2.1
18/04/23 16:24:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/04/23 16:24:49 WARN Utils: Your hostname, ecs-909c resolves to a loopback address: 127.0.0.1; using 192.168.0.206 instead (on interface eth0)
18/04/23 16:24:49 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
18/04/23 16:24:50 INFO SparkContext: Submitted application: totalNum: 10000000 threadNum: 16 taskNum: 100 resultIsEmpty: true file path: /tmp/carbondata runInLocal: true generateFile: true deleteFile: false
18/04/23 16:24:50 INFO SecurityManager: Changing view acls to: root
18/04/23 16:24:50 INFO SecurityManager: Changing modify acls to: root
18/04/23 16:24:50 INFO SecurityManager: Changing view acls groups to:
18/04/23 16:24:50 INFO SecurityManager: Changing modify acls groups to:
18/04/23 16:24:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
18/04/23 16:24:50 INFO Utils: Successfully started service 'sparkDriver' on port 35444.
18/04/23 16:24:50 INFO SparkEnv: Registering MapOutputTracker
18/04/23 16:24:50 INFO SparkEnv: Registering BlockManagerMaster
18/04/23 16:24:50 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/04/23 16:24:50 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/04/23 16:24:50 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-186ff215-cd17-4b23-9fe0-c55e169f5d87
18/04/23 16:24:50 INFO MemoryStore: MemoryStore started with capacity 1957.8 MB
18/04/23 16:24:50 INFO SparkEnv: Registering OutputCommitCoordinator
18/04/23 16:24:50 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/04/23 16:24:50 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.206:4040
18/04/23 16:24:50 INFO Executor: Starting executor ID driver on host localhost
18/04/23 16:24:50 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46504.
18/04/23 16:24:50 INFO NettyBlockTransferService: Server created on 192.168.0.206:46504
18/04/23 16:24:50 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/04/23 16:24:50 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.206, 46504, None)
18/04/23 16:24:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.206:46504 with 1957.8 MB RAM, BlockManagerId(driver, 192.168.0.206, 46504, None)
18/04/23 16:24:50 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.206, 46504, None)
18/04/23 16:24:50 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.206, 46504, None)
18/04/23 16:24:51 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/huawei/xubo/git/carbondata1/spark-warehouse').
18/04/23 16:24:51 INFO SharedState: Warehouse path is 'file:/huawei/xubo/git/carbondata1/spark-warehouse'.
18/04/23 16:24:51 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
18/04/23 16:24:52 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/04/23 16:24:52 INFO ObjectStore: ObjectStore, initialize called
18/04/23 16:24:52 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/04/23 16:24:52 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
18/04/23 16:24:53 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/04/23 16:24:54 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/04/23 16:24:54 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/04/23 16:24:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/04/23 16:24:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/04/23 16:24:55 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
18/04/23 16:24:55 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/04/23 16:24:55 INFO ObjectStore: Initialized ObjectStore
18/04/23 16:24:55 INFO HiveMetaStore: Added admin role in metastore
18/04/23 16:24:55 INFO HiveMetaStore: Added public role in metastore
18/04/23 16:24:55 INFO HiveMetaStore: No user is added in admin role, since config is empty
18/04/23 16:24:55 INFO HiveMetaStore: 0: get_all_databases
18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_all_databases
18/04/23 16:24:55 INFO HiveMetaStore: 0: get_functions: db=default pat=*
18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
18/04/23 16:24:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
18/04/23 16:24:55 INFO SessionState: Created local directory: /tmp/a2a6b2d9-89b8-49ed-a1e4-510d33e89963_resources
18/04/23 16:24:55 INFO SessionState: Created HDFS directory: /tmp/hive/root/a2a6b2d9-89b8-49ed-a1e4-510d33e89963
18/04/23 16:24:55 INFO SessionState: Created local directory: /tmp/root/a2a6b2d9-89b8-49ed-a1e4-510d33e89963
18/04/23 16:24:55 INFO SessionState: Created HDFS directory: /tmp/hive/root/a2a6b2d9-89b8-49ed-a1e4-510d33e89963/_tmp_space.db
18/04/23 16:24:55 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/huawei/xubo/git/carbondata1/spark-warehouse
18/04/23 16:24:55 INFO HiveMetaStore: 0: get_database: default
18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_database: default
18/04/23 16:24:55 INFO HiveMetaStore: 0: get_database: global_temp
18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_database: global_temp
18/04/23 16:24:55 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
18/04/23 16:24:56 INFO SessionState: Created local directory: /tmp/fe452d60-a43e-4edf-8945-600c341bb02c_resources
18/04/23 16:24:56 INFO SessionState: Created HDFS directory: /tmp/hive/root/fe452d60-a43e-4edf-8945-600c341bb02c
18/04/23 16:24:56 INFO SessionState: Created local directory: /tmp/root/fe452d60-a43e-4edf-8945-600c341bb02c
18/04/23 16:24:56 INFO SessionState: Created HDFS directory: /tmp/hive/root/fe452d60-a43e-4edf-8945-600c341bb02c/_tmp_space.db
18/04/23 16:24:56 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/huawei/xubo/git/carbondata1/spark-warehouse
18/04/23 16:24:56 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
Environment information:
spark.master: local[8]
spark.driver.cores: default value
spark.driver.memory: default value
spark.executor.cores: default value
spark.executor.memory: default value
spark.executor.instances: default value
SPARK_VERSION:2.2.1
CARBONDATA_VERSION:1.4.0-SNAPSHOT
Parameters information:
totalNum: 10000000 threadNum: 16 taskNum: 100 resultIsEmpty: true file path: /tmp/carbondata runInLocal: true generateFile: true deleteFile: false
Start generate 10000000 records, schema: StructType(StructField(id,StringType,false), StructField(city,StringType,false), StructField(country,StringType,false), StructField(planet,StringType,false), StructField(m1,ShortType,false), StructField(m2,IntegerType,false), StructField(m3,LongType,false), StructField(m4,DoubleType,false), StructField(m5,DecimalType(30,10),false))
Num10000000_comparetest_parquet completed, time: 33.757 sec
18/04/23 16:25:37 AUDIT CarbonCreateTableCommand: [ecs-909c][root][Thread-1]Creating Table with Database name [default] and Table name [num10000000_comparetest_carbonv3]
18/04/23 16:25:38 AUDIT CarbonCreateTableCommand: [ecs-909c][root][Thread-1]Table created with Database name [default] and Table name [num10000000_comparetest_carbonv3]
18/04/23 16:25:38 AUDIT CarbonDataRDDFactory$: [ecs-909c][root][Thread-1]Data load request has been received for table default.num10000000_comparetest_carbonv3
18/04/23 16:26:36 AUDIT CarbonDataRDDFactory$: [ecs-909c][root][Thread-1]Data load is successful for default.num10000000_comparetest_carbonv3
Num10000000_comparetest_carbonV3 completed, time: 59.577 sec
Start running queries for Num10000000_comparetest_carbonV3...
Min: min time Max: max time 90%: 90% time 99%: 99% time Avg: average time Count: number of result Query X: running different query sql Result: show it when ResultIsEmpty is false Total execute time: total runtime
18/04/23 16:27:12 ERROR Inbox: Ignoring error
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:75)
at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.<init>(ChunkRowIterator.java:40)
at org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:84)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleRequest(SearchRequestHandler.java:95)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleSearch(SearchRequestHandler.java:62)
at org.apache.spark.search.Searcher$$anonfun$receiveAndReply$1.applyOrElse(Searcher.scala:41)
at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:105)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:216)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:71)
... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.carbondata.core.datastore.impl.FileReaderImpl.readByteBuffer(FileReaderImpl.java:200)
at org.apache.carbondata.core.datastore.chunk.reader.measure.v3.CompressedMeasureChunkFileBasedReaderV3.readRawMeasureChunksInGroup(CompressedMeasureChunkFileBasedReaderV3.java:157)
at org.apache.carbondata.core.datastore.chunk.reader.measure.AbstractMeasureChunkReaderV2V3Format.readRawMeasureChunks(AbstractMeasureChunkReaderV2V3Format.java:77)
at org.apache.carbondata.core.indexstore.blockletindex.BlockletDataRefNode.readMeasureChunks(BlockletDataRefNode.java:147)
at org.apache.carbondata.core.scan.scanner.impl.BlockletFilterScanner.executeFilter(BlockletFilterScanner.java:278)
at org.apache.carbondata.core.scan.scanner.impl.BlockletFilterScanner.scanBlocklet(BlockletFilterScanner.java:101)
at org.apache.carbondata.core.scan.processor.BlockScan.scan(BlockScan.java:69)
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator$1.call(AbstractSearchModeResultIterator.java:59)
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator$1.call(AbstractSearchModeResultIterator.java:53)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
18/04/23 16:27:12 ERROR Inbox: Ignoring error
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:75)
at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.<init>(ChunkRowIterator.java:40)
at org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:84)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleRequest(SearchRequestHandler.java:95)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleSearch(SearchRequestHandler.java:62)
at org.apache.spark.search.Searcher$$anonfun$receiveAndReply$1.applyOrElse(Searcher.scala:41)
at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:105)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:216)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:71)
... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
18/04/23 16:27:27 ERROR Inbox: Ignoring error
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:75)
at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.<init>(ChunkRowIterator.java:40)
at org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:84)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleRequest(SearchRequestHandler.java:95)
at org.apache.carbondata.store.worker.SearchRequestHandler.handleSearch(SearchRequestHandler.java:62)
at org.apache.spark.search.Searcher$$anonfun$receiveAndReply$1.applyOrElse(Searcher.scala:41)
at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:105)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:216)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:71)
... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
18/04/23 16:27:27 ERROR CarbonSession: Exception when executing search mode: Exception thrown in awaitResult: , fallback to SparkSQL
18/04/23 16:27:27 ERROR CarbonSession: Exception when executing search mode: Exception thrown in awaitResult: , fallback to SparkSQL
18/04/23 16:27:28 ERROR CarbonSession: Exception when executing search mode: Exception thrown in awaitResult: , fallback to SparkSQL
{code}
> OutOfMemoryError when use search mode in ConcurrentQueryBenchmark
> -----------------------------------------------------------------
>
> Key: CARBONDATA-2382
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2382
> Project: CarbonData
> Issue Type: Bug
> Reporter: xubo245
> Assignee: xubo245
> Priority: Major
>
> {code:java}
> 18/04/23 16:24:49 WARN Utils: Your hostname, ecs-909c resolves to a loopback address: 127.0.0.1; using 192.168.0.206 instead (on interface eth0)
> 18/04/23 16:24:49 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
> 18/04/23 16:24:50 INFO SparkContext: Submitted application: totalNum: 10000000 threadNum: 16 taskNum: 100 resultIsEmpty: true file path: /tmp/carbondata runInLocal: true generateFile: true deleteFile: false
> 18/04/23 16:24:50 INFO SecurityManager: Changing view acls to: root
> 18/04/23 16:24:50 INFO SecurityManager: Changing modify acls to: root
> 18/04/23 16:24:50 INFO SecurityManager: Changing view acls groups to:
> 18/04/23 16:24:50 INFO SecurityManager: Changing modify acls groups to:
> 18/04/23 16:24:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
> 18/04/23 16:24:50 INFO Utils: Successfully started service 'sparkDriver' on port 35444.
> 18/04/23 16:24:50 INFO SparkEnv: Registering MapOutputTracker
> 18/04/23 16:24:50 INFO SparkEnv: Registering BlockManagerMaster
> 18/04/23 16:24:50 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
> 18/04/23 16:24:50 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
> 18/04/23 16:24:50 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-186ff215-cd17-4b23-9fe0-c55e169f5d87
> 18/04/23 16:24:50 INFO MemoryStore: MemoryStore started with capacity 1957.8 MB
> 18/04/23 16:24:50 INFO SparkEnv: Registering OutputCommitCoordinator
> 18/04/23 16:24:50 INFO Utils: Successfully started service 'SparkUI' on port 4040.
> 18/04/23 16:24:50 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.0.206:4040
> 18/04/23 16:24:50 INFO Executor: Starting executor ID driver on host localhost
> 18/04/23 16:24:50 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46504.
> 18/04/23 16:24:50 INFO NettyBlockTransferService: Server created on 192.168.0.206:46504
> 18/04/23 16:24:50 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
> 18/04/23 16:24:50 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.206, 46504, None)
> 18/04/23 16:24:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.206:46504 with 1957.8 MB RAM, BlockManagerId(driver, 192.168.0.206, 46504, None)
> 18/04/23 16:24:50 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.206, 46504, None)
> 18/04/23 16:24:50 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.206, 46504, None)
> 18/04/23 16:24:51 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/huawei/xubo/git/carbondata1/spark-warehouse').
> 18/04/23 16:24:51 INFO SharedState: Warehouse path is 'file:/huawei/xubo/git/carbondata1/spark-warehouse'.
> 18/04/23 16:24:51 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
> 18/04/23 16:24:52 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
> 18/04/23 16:24:52 INFO ObjectStore: ObjectStore, initialize called
> 18/04/23 16:24:52 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 18/04/23 16:24:52 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
> 18/04/23 16:24:53 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 18/04/23 16:24:54 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
> 18/04/23 16:24:54 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
> 18/04/23 16:24:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
> 18/04/23 16:24:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
> 18/04/23 16:24:55 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
> 18/04/23 16:24:55 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
> 18/04/23 16:24:55 INFO ObjectStore: Initialized ObjectStore
> 18/04/23 16:24:55 INFO HiveMetaStore: Added admin role in metastore
> 18/04/23 16:24:55 INFO HiveMetaStore: Added public role in metastore
> 18/04/23 16:24:55 INFO HiveMetaStore: No user is added in admin role, since config is empty
> 18/04/23 16:24:55 INFO HiveMetaStore: 0: get_all_databases
> 18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_all_databases
> 18/04/23 16:24:55 INFO HiveMetaStore: 0: get_functions: db=default pat=*
> 18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_functions: db=default pat=*
> 18/04/23 16:24:55 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
> 18/04/23 16:24:55 INFO SessionState: Created local directory: /tmp/a2a6b2d9-89b8-49ed-a1e4-510d33e89963_resources
> 18/04/23 16:24:55 INFO SessionState: Created HDFS directory: /tmp/hive/root/a2a6b2d9-89b8-49ed-a1e4-510d33e89963
> 18/04/23 16:24:55 INFO SessionState: Created local directory: /tmp/root/a2a6b2d9-89b8-49ed-a1e4-510d33e89963
> 18/04/23 16:24:55 INFO SessionState: Created HDFS directory: /tmp/hive/root/a2a6b2d9-89b8-49ed-a1e4-510d33e89963/_tmp_space.db
> 18/04/23 16:24:55 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/huawei/xubo/git/carbondata1/spark-warehouse
> 18/04/23 16:24:55 INFO HiveMetaStore: 0: get_database: default
> 18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_database: default
> 18/04/23 16:24:55 INFO HiveMetaStore: 0: get_database: global_temp
> 18/04/23 16:24:55 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_database: global_temp
> 18/04/23 16:24:55 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
> 18/04/23 16:24:56 INFO SessionState: Created local directory: /tmp/fe452d60-a43e-4edf-8945-600c341bb02c_resources
> 18/04/23 16:24:56 INFO SessionState: Created HDFS directory: /tmp/hive/root/fe452d60-a43e-4edf-8945-600c341bb02c
> 18/04/23 16:24:56 INFO SessionState: Created local directory: /tmp/root/fe452d60-a43e-4edf-8945-600c341bb02c
> 18/04/23 16:24:56 INFO SessionState: Created HDFS directory: /tmp/hive/root/fe452d60-a43e-4edf-8945-600c341bb02c/_tmp_space.db
> 18/04/23 16:24:56 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/huawei/xubo/git/carbondata1/spark-warehouse
> 18/04/23 16:24:56 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
> Environment information:
> spark.master: local[8]
> spark.driver.cores: default value
> spark.driver.memory: default value
> spark.executor.cores: default value
> spark.executor.memory: default value
> spark.executor.instances: default value
> SPARK_VERSION:2.2.1
> CARBONDATA_VERSION:1.4.0-SNAPSHOT
> Parameters information:
> totalNum: 10000000 threadNum: 16 taskNum: 100 resultIsEmpty: true file path: /tmp/carbondata runInLocal: true generateFile: true deleteFile: false
> Start generate 10000000 records, schema: StructType(StructField(id,StringType,false), StructField(city,StringType,false), StructField(country,StringType,false), StructField(planet,StringType,false), StructField(m1,ShortType,false), StructField(m2,IntegerType,false), StructField(m3,LongType,false), StructField(m4,DoubleType,false), StructField(m5,DecimalType(30,10),false))
> Num10000000_comparetest_parquet completed, time: 33.757 sec
> 18/04/23 16:25:37 AUDIT CarbonCreateTableCommand: [ecs-909c][root][Thread-1]Creating Table with Database name [default] and Table name [num10000000_comparetest_carbonv3]
> 18/04/23 16:25:38 AUDIT CarbonCreateTableCommand: [ecs-909c][root][Thread-1]Table created with Database name [default] and Table name [num10000000_comparetest_carbonv3]
> 18/04/23 16:25:38 AUDIT CarbonDataRDDFactory$: [ecs-909c][root][Thread-1]Data load request has been received for table default.num10000000_comparetest_carbonv3
> 18/04/23 16:26:36 AUDIT CarbonDataRDDFactory$: [ecs-909c][root][Thread-1]Data load is successful for default.num10000000_comparetest_carbonv3
> Num10000000_comparetest_carbonV3 completed, time: 59.577 sec
> Start running queries for Num10000000_comparetest_carbonV3...
> Min: min time Max: max time 90%: 90% time 99%: 99% time Avg: average time Count: number of result Query X: running different query sql Result: show it when ResultIsEmpty is false Total execute time: total runtime
> 18/04/23 16:27:12 ERROR Inbox: Ignoring error
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:75)
> at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.<init>(ChunkRowIterator.java:40)
> at org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:84)
> at org.apache.carbondata.store.worker.SearchRequestHandler.handleRequest(SearchRequestHandler.java:95)
> at org.apache.carbondata.store.worker.SearchRequestHandler.handleSearch(SearchRequestHandler.java:62)
> at org.apache.spark.search.Searcher$$anonfun$receiveAndReply$1.applyOrElse(Searcher.scala:41)
> at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:105)
> at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
> at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
> at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:216)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:71)
> ... 12 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> at org.apache.carbondata.core.datastore.impl.FileReaderImpl.readByteBuffer(FileReaderImpl.java:200)
> at org.apache.carbondata.core.datastore.chunk.reader.measure.v3.CompressedMeasureChunkFileBasedReaderV3.readRawMeasureChunksInGroup(CompressedMeasureChunkFileBasedReaderV3.java:157)
> at org.apache.carbondata.core.datastore.chunk.reader.measure.AbstractMeasureChunkReaderV2V3Format.readRawMeasureChunks(AbstractMeasureChunkReaderV2V3Format.java:77)
> at org.apache.carbondata.core.indexstore.blockletindex.BlockletDataRefNode.readMeasureChunks(BlockletDataRefNode.java:147)
> at org.apache.carbondata.core.scan.scanner.impl.BlockletFilterScanner.executeFilter(BlockletFilterScanner.java:278)
> at org.apache.carbondata.core.scan.scanner.impl.BlockletFilterScanner.scanBlocklet(BlockletFilterScanner.java:101)
> at org.apache.carbondata.core.scan.processor.BlockScan.scan(BlockScan.java:69)
> at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator$1.call(AbstractSearchModeResultIterator.java:59)
> at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator$1.call(AbstractSearchModeResultIterator.java:53)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> 18/04/23 16:27:12 ERROR Inbox: Ignoring error
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:75)
> at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.<init>(ChunkRowIterator.java:40)
> at org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:84)
> at org.apache.carbondata.store.worker.SearchRequestHandler.handleRequest(SearchRequestHandler.java:95)
> at org.apache.carbondata.store.worker.SearchRequestHandler.handleSearch(SearchRequestHandler.java:62)
> at org.apache.spark.search.Searcher$$anonfun$receiveAndReply$1.applyOrElse(Searcher.scala:41)
> at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:105)
> at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
> at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
> at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:216)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:71)
> ... 12 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 18/04/23 16:27:27 ERROR Inbox: Ignoring error
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
> at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:75)
> at org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.<init>(ChunkRowIterator.java:40)
> at org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:84)
> at org.apache.carbondata.store.worker.SearchRequestHandler.handleRequest(SearchRequestHandler.java:95)
> at org.apache.carbondata.store.worker.SearchRequestHandler.handleSearch(SearchRequestHandler.java:62)
> at org.apache.spark.search.Searcher$$anonfun$receiveAndReply$1.applyOrElse(Searcher.scala:41)
> at org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:105)
> at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)
> at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)
> at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:216)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.carbondata.core.scan.result.iterator.AbstractSearchModeResultIterator.hasNext(AbstractSearchModeResultIterator.java:71)
> ... 12 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 18/04/23 16:27:27 ERROR CarbonSession: Exception when executing search mode: Exception thrown in awaitResult: , fallback to SparkSQL
> 18/04/23 16:27:27 ERROR CarbonSession: Exception when executing search mode: Exception thrown in awaitResult: , fallback to SparkSQL
> 18/04/23 16:27:28 ERROR CarbonSession: Exception when executing search mode: Exception thrown in awaitResult: , fallback to SparkSQL
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)