You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by xatax <ro...@gmail.com> on 2021/02/24 19:04:00 UTC
Cube migration across hadoop clusters
Per: http://kylin.apache.org/docs/tutorial/cube_migration.html
It says: "Support to migrate cube data (segments data on HBase) from QA env
to PROD env."
I tried the migration but it only migrates the cube metadata without the
data. The source kylin instance has the cube buiilt and in ready state.
After I executed the command it said it was successful. But when I checked
the destination kylin instance it only got the cube definition and not the
data.
Can you please clarify if migrating the data is possible or not?
Or is the migration of the data a separate activity which involves the hbase
snapshot import export.
thanks in advance
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: Cube migration across hadoop clusters
Posted by xatax <ro...@gmail.com>.
Thanks Zhichao,
I tried that and am now getting this:
[hadoop@ip-172-31-81-255 apache-kylin-3.1.0-bin-hbase1x]$
[hadoop@ip-172-31-81-255 apache-kylin-3.1.0-bin-hbase1x]$ ./bin/kylin.sh
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI -kylinUriSrc
ADMIN:KYLIN@ip-172-31-81-255.ec2.internal:7070 -kylinUriDst
ADMIN:KYLIN@ip-172-31-91-198.ec2.internal:7070 -cube kylin_sales_cube
-execute true -overwrite true -schemaOnly false
Using cached dependency...
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support
was removed in 8.0
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x/tool/kylin-tool-3.1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Running org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI
-kylinUriSrc ADMIN:KYLIN@ip-172-31-81-255.ec2.internal:7070 -kylinUriDst
ADMIN:KYLIN@ip-172-31-91-198.ec2.internal:7070 -cube kylin_sales_cube
-execute true -overwrite true -schemaOnly false
2021-02-26 10:06:54,000 INFO [main] common.KylinConfig:118 : Loading
kylin-defaults.properties from
file:/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x/tool/kylin-tool-3.1.0.jar!/kylin-defaults.properties
2021-02-26 10:06:54,028 DEBUG [main] common.KylinConfig:346 : KYLIN_CONF
property was not set, will seek KYLIN_HOME env variable
2021-02-26 10:06:54,029 INFO [main] common.KylinConfig:352 : Use
KYLIN_HOME=/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x
2021-02-26 10:06:54,031 INFO [main] common.KylinConfig:153 : Initialized a
new KylinConfig from getInstanceFromEnv : 245765246
2021-02-26 10:06:56,326 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x545607f2 connecting to ZooKeeper
ensemble=ip-172-31-81-255.ec2.internal:2181
2021-02-26 10:06:56,331 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:zookeeper.version=3.4.14-5f788d5e8f90539ee331702c753fa250727128f4,
built on 12/14/2019 00:35 GMT
2021-02-26 10:06:56,331 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:host.name=ip-172-31-81-255.ec2.internal
2021-02-26 10:06:56,332 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.version=1.8.0_272
2021-02-26 10:06:56,332 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.vendor=Red Hat, Inc.
2021-02-26 10:06:56,332 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.56.amzn1.x86_64/jre
2021-02-26 10:06:56,332 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.class.path= /*removed*/
2021-02-26 10:06:56,333 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.library.path=:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native
2021-02-26 10:06:56,333 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.io.tmpdir=/tmp
2021-02-26 10:06:56,334 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.compiler=<NA>
2021-02-26 10:06:56,334 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.name=Linux
2021-02-26 10:06:56,334 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.arch=amd64
2021-02-26 10:06:56,335 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.version=4.14.154-99.181.amzn1.x86_64
2021-02-26 10:06:56,335 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.name=hadoop
2021-02-26 10:06:56,335 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.home=/home/hadoop
2021-02-26 10:06:56,335 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.dir=/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x
2021-02-26 10:06:56,336 INFO [main] zookeeper.ZooKeeper:442 : Initiating
client connection, connectString=ip-172-31-81-255.ec2.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7207cb51
2021-02-26 10:06:56,357 INFO
[main-SendThread(ip-172-31-81-255.ec2.internal:2181)]
zookeeper.ClientCnxn:1025 : Opening socket connection to server
ip-172-31-81-255.ec2.internal/172.31.81.255:2181. Will not attempt to
authenticate using SASL (unknown error)
2021-02-26 10:06:56,363 INFO
[main-SendThread(ip-172-31-81-255.ec2.internal:2181)]
zookeeper.ClientCnxn:879 : Socket connection established to
ip-172-31-81-255.ec2.internal/172.31.81.255:2181, initiating session
2021-02-26 10:06:56,369 INFO
[main-SendThread(ip-172-31-81-255.ec2.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-81-255.ec2.internal/172.31.81.255:2181, sessionid =
0x1000002a32a0020, negotiated timeout = 40000
2021-02-26 10:06:56,878 DEBUG [main] hbase.HBaseConnection:316 : HTable
'kylin_metadata' already exists
2021-02-26 10:06:57,222 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.TableMetadataManager
2021-02-26 10:06:57,229 INFO [main] persistence.ResourceStore:90 : Using
metadata url kylin_metadata@hbase for resource store
2021-02-26 10:06:57,386 DEBUG [main] hbase.HBaseConnection:181 : Using the
working dir FS for HBase: hdfs://ip-172-31-81-255.ec2.internal:8020
2021-02-26 10:06:57,386 INFO [main] hbase.HBaseConnection:267 : connection
is null or closed, creating a new one
2021-02-26 10:06:57,387 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x10cd6753 connecting to ZooKeeper
ensemble=ip-172-31-81-255.ec2.internal:2181
2021-02-26 10:06:57,387 INFO [main] zookeeper.ZooKeeper:442 : Initiating
client connection, connectString=ip-172-31-81-255.ec2.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@71ad3d8a
2021-02-26 10:06:57,388 INFO
[main-SendThread(ip-172-31-81-255.ec2.internal:2181)]
zookeeper.ClientCnxn:1025 : Opening socket connection to server
ip-172-31-81-255.ec2.internal/172.31.81.255:2181. Will not attempt to
authenticate using SASL (unknown error)
2021-02-26 10:06:57,389 INFO
[main-SendThread(ip-172-31-81-255.ec2.internal:2181)]
zookeeper.ClientCnxn:879 : Socket connection established to
ip-172-31-81-255.ec2.internal/172.31.81.255:2181, initiating session
2021-02-26 10:06:57,392 INFO
[main-SendThread(ip-172-31-81-255.ec2.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-81-255.ec2.internal/172.31.81.255:2181, sessionid =
0x1000002a32a0021, negotiated timeout = 40000
2021-02-26 10:06:57,407 DEBUG [main] hbase.HBaseConnection:316 : HTable
'kylin_metadata' already exists
2021-02-26 10:06:57,450 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading TableDesc from kylin_metadata(key='/table')@kylin_metadata@hbase
2021-02-26 10:06:57,528 INFO [main] measure.MeasureTypeFactory:121 :
Checking custom measure types from kylin config
2021-02-26 10:06:57,529 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(hllc), class
org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
2021-02-26 10:06:57,534 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(bitmap), class
org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
2021-02-26 10:06:57,536 INFO [main] measure.MeasureTypeFactory:150 :
registering TOP_N(topn), class
org.apache.kylin.measure.topn.TopNMeasureType$Factory
2021-02-26 10:06:57,538 INFO [main] measure.MeasureTypeFactory:150 :
registering RAW(raw), class
org.apache.kylin.measure.raw.RawMeasureType$Factory
2021-02-26 10:06:57,539 INFO [main] measure.MeasureTypeFactory:150 :
registering EXTENDED_COLUMN(extendedcolumn), class
org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
2021-02-26 10:06:57,540 INFO [main] measure.MeasureTypeFactory:150 :
registering PERCENTILE_APPROX(percentile), class
org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
2021-02-26 10:06:57,541 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(dim_dc), class
org.apache.kylin.measure.dim.DimCountDistinctMeasureType$Factory
2021-02-26 10:06:57,542 INFO [main] measure.MeasureTypeFactory:150 :
registering STDDEV_SUM(stddev_sum), class
org.apache.kylin.measure.stddev.StdDevSumMeasureType$Factory
2021-02-26 10:06:57,546 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(bitmap_map), class
org.apache.kylin.measure.map.bitmap.BitmapMapMeasureType$Factory
2021-02-26 10:06:57,548 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
6 TableDesc(s) out of 6 resource with 0 errors
2021-02-26 10:06:57,550 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2021-02-26 10:06:57,551 DEBUG [main] cachesync.Broadcaster:102 : 1 nodes in
the cluster: [localhost:7070]
2021-02-26 10:06:57,553 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading TableExtDesc from
kylin_metadata(key='/table_exd')@kylin_metadata@hbase
2021-02-26 10:06:57,556 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 TableExtDesc(s) out of 0 resource with 0 errors
2021-02-26 10:06:57,557 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading ExternalFilterDesc from
kylin_metadata(key='/ext_filter')@kylin_metadata@hbase
2021-02-26 10:06:57,558 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 ExternalFilterDesc(s) out of 0 resource with 0 errors
2021-02-26 10:06:57,561 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.model.DataModelManager
2021-02-26 10:06:57,562 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading DataModelDesc from
kylin_metadata(key='/model_desc')@kylin_metadata@hbase
2021-02-26 10:06:57,588 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.project.ProjectManager
2021-02-26 10:06:57,588 INFO [main] project.ProjectManager:85 :
Initializing ProjectManager with metadata url kylin_metadata@hbase
2021-02-26 10:06:57,589 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading ProjectInstance from
kylin_metadata(key='/project')@kylin_metadata@hbase
2021-02-26 10:06:57,602 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
1 ProjectInstance(s) out of 1 resource with 0 errors
2021-02-26 10:06:57,608 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
2 DataModelDesc(s) out of 2 resource with 0 errors
2021-02-26 10:06:57,610 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.storage.hybrid.HybridManager
2021-02-26 10:06:57,610 INFO [main] hybrid.HybridManager:71 : Initializing
HybridManager with config kylin_metadata@hbase
2021-02-26 10:06:57,610 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading HybridInstance from
kylin_metadata(key='/hybrid')@kylin_metadata@hbase
2021-02-26 10:06:57,612 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 HybridInstance(s) out of 0 resource with 0 errors
2021-02-26 10:06:57,616 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.cube.CubeManager
2021-02-26 10:06:57,622 INFO [main] cube.CubeManager:136 : Initializing
CubeManager with config kylin_metadata@hbase
2021-02-26 10:06:57,623 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading CubeInstance from kylin_metadata(key='/cube')@kylin_metadata@hbase
2021-02-26 10:06:57,659 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.cube.CubeDescManager
2021-02-26 10:06:57,659 INFO [main] cube.CubeDescManager:91 : Initializing
CubeDescManager with config kylin_metadata@hbase
2021-02-26 10:06:57,660 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading CubeDesc from
kylin_metadata(key='/cube_desc')@kylin_metadata@hbase
2021-02-26 10:06:57,707 INFO [main] model.RowKeyColDesc:99 : Implicitly
convert encoding to time
2021-02-26 10:06:57,707 INFO [main] model.RowKeyColDesc:99 : Implicitly
convert encoding to time
2021-02-26 10:06:57,707 INFO [main] model.RowKeyColDesc:95 : Implicitly
convert encoding to date
2021-02-26 10:06:57,708 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
2 CubeDesc(s) out of 2 resource with 0 errors
2021-02-26 10:06:57,708 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
2 CubeInstance(s) out of 2 resource with 0 errors
2021-02-26 10:06:57,709 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class
org.apache.kylin.metadata.realization.RealizationRegistry
2021-02-26 10:06:57,709 INFO [main] realization.RealizationRegistry:54 :
Initializing RealizationRegistry with metadata url kylin_metadata@hbase
2021-02-26 10:06:57,709 INFO [main] realization.RealizationRegistry:81 :
RealizationRegistry is
{HYBRID=org.apache.kylin.storage.hybrid.HybridManager@4c58255,
CUBE=org.apache.kylin.cube.CubeManager@eac3a26}
2021-02-26 10:06:57,710 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.dict.DictionaryManager
2021-02-26 10:06:57,711 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.dict.lookup.SnapshotManager
2021-02-26 10:06:58,164 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x1e1e9ef3 connecting to ZooKeeper
ensemble=ip-172-31-91-198.ec2.internal:2181
2021-02-26 10:06:58,165 INFO [main] zookeeper.ZooKeeper:442 : Initiating
client connection, connectString=ip-172-31-91-198.ec2.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@3dd31157
2021-02-26 10:06:58,166 INFO
[main-SendThread(ip-172-31-91-198.ec2.internal:2181)]
zookeeper.ClientCnxn:1025 : Opening socket connection to server
ip-172-31-91-198.ec2.internal/172.31.91.198:2181. Will not attempt to
authenticate using SASL (unknown error)
2021-02-26 10:06:58,167 INFO
[main-SendThread(ip-172-31-91-198.ec2.internal:2181)]
zookeeper.ClientCnxn:879 : Socket connection established to
ip-172-31-91-198.ec2.internal/172.31.91.198:2181, initiating session
2021-02-26 10:06:58,169 INFO
[main-SendThread(ip-172-31-91-198.ec2.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-91-198.ec2.internal/172.31.91.198:2181, sessionid =
0x10000026cd90019, negotiated timeout = 40000
2021-02-26 10:06:58,202 DEBUG [main] hbase.HBaseConnection:316 : HTable
'kylin_metadata' already exists
2021-02-26 10:06:58,209 INFO [main]
migration.CubeMigrationCrossClusterCLI:327 : start to migrate cube
CUBE[name=kylin_sales_cube]
2021-02-26 10:07:01,164 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_CAL_DT.json
2021-02-26 10:07:01,165 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_CAL_DT', columns=/*removed*/
2021-02-26 10:07:01,169 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_CATEGORY_GROUPINGS.json
2021-02-26 10:07:01,169 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_CATEGORY_GROUPINGS', columns=/*removed*/
2021-02-26 10:07:01,172 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_ACCOUNT.json
2021-02-26 10:07:01,172 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_ACCOUNT', columns=/*removed*/
2021-02-26 10:07:01,174 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_COUNTRY.json
2021-02-26 10:07:01,174 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_COUNTRY', columns=/*removed*/
2021-02-26 10:07:01,177 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_SALES.json
2021-02-26 10:07:01,177 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_SALES', columns=/*removed*/
2021-02-26 10:07:01,180 INFO [main] migration.DstClusterUtil:249 : saved
resource /model_desc/kylin_sales_model.json
2021-02-26 10:07:01,180 INFO [main] migration.DstClusterUtil:170 : saved
model DataModelDesc [name=kylin_sales_model]
2021-02-26 10:07:01,198 INFO [main] migration.DstClusterUtil:249 : saved
resource /cube_desc/kylin_sales_cube.json
2021-02-26 10:07:01,198 INFO [main] migration.DstClusterUtil:178 : saved
cube desc CubeDesc [name=kylin_sales_cube]
2021-02-26 10:07:01,200 INFO [main]
migration.CubeMigrationCrossClusterCLI:485 : start to migrate segment:
CUBE[name=kylin_sales_cube] 20120101000000_20120701000000
2021-02-26 10:07:01,214 INFO [main] migration.DstClusterUtil:256 : saved
resource
/cube_statistics/kylin_sales_cube/f836913c-4bb7-e8ab-907d-70efc78f9f08.seq
2021-02-26 10:07:01,216 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_COUNTRY/e5af18b4-3e78-8acd-0549-c75d868bcedd.dict
has already existed in destination cluster
2021-02-26 10:07:01,217 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/CATEG_LVL3_NAME/8668e511-36d6-24cf-4c3f-ec0051f1fd03.dict
has already existed in destination cluster
2021-02-26 10:07:01,218 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_SALES/SELLER_ID/08a91346-5d12-ad4f-3ed7-0fd2330bedb9.dict
has already existed in destination cluster
2021-02-26 10:07:01,219 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_SALES/LSTG_SITE_ID/1d0b222c-8600-6874-8c43-9c730500061c.dict
has already existed in destination cluster
2021-02-26 10:07:01,220 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_SALES/OPS_REGION/1837f2a0-dc07-9094-01e4-69140c9c602a.dict
has already existed in destination cluster
2021-02-26 10:07:01,221 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_SALES/LEAF_CATEG_ID/c9184a84-7a75-4136-8b59-459c977c67e0.dict
has already existed in destination cluster
2021-02-26 10:07:01,222 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_COUNTRY/NAME/14682a87-1854-d657-6609-44a1c06fbf50.dict
has already existed in destination cluster
2021-02-26 10:07:01,223 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_COUNTRY/e5af18b4-3e78-8acd-0549-c75d868bcedd.dict
has already existed in destination cluster
2021-02-26 10:07:01,224 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_SALES/LSTG_FORMAT_NAME/ffedd2e1-9145-d8a3-07f1-a0f925a23627.dict
has already existed in destination cluster
2021-02-26 10:07:01,225 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_COUNTRY/NAME/14682a87-1854-d657-6609-44a1c06fbf50.dict
has already existed in destination cluster
2021-02-26 10:07:01,226 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_BUYER_LEVEL/7dd3e83e-5061-aa49-c2f1-5f405afa7e2b.dict
has already existed in destination cluster
2021-02-26 10:07:01,227 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_ACCOUNT/ACCOUNT_SELLER_LEVEL/32a2dfd4-6f87-c00c-ef28-4272f9bc017c.dict
has already existed in destination cluster
2021-02-26 10:07:01,228 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_SALES/OPS_USER_ID/1280658b-9adb-f8eb-83a2-60dfb729a6d5.dict
has already existed in destination cluster
2021-02-26 10:07:01,229 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/META_CATEG_NAME/57228867-3388-54d4-c560-ddce4ccc456f.dict
has already existed in destination cluster
2021-02-26 10:07:01,230 INFO [main]
migration.CubeMigrationCrossClusterCLI:571 : Item
/dict/DEFAULT.KYLIN_CATEGORY_GROUPINGS/CATEG_LVL2_NAME/7a8f9706-ecf2-eca0-07c7-1d757a261062.dict
has already existed in destination cluster
2021-02-26 10:07:01,230 INFO [main]
migration.CubeMigrationCrossClusterCLI:590 : Item
/table_snapshot/DEFAULT.KYLIN_CAL_DT/c2b99839-f1f7-0fa3-6d0a-92312e36d993.snapshot
has already existed in destination cluster
2021-02-26 10:07:01,231 INFO [main]
migration.CubeMigrationCrossClusterCLI:590 : Item
/table_snapshot/DEFAULT.KYLIN_CATEGORY_GROUPINGS/f1859dd6-f508-9cae-8796-1e98a64ac743.snapshot
has already existed in destination cluster
2021-02-26 10:07:01,232 INFO [main]
migration.CubeMigrationCrossClusterCLI:590 : Item
/table_snapshot/DEFAULT.KYLIN_ACCOUNT/f95df070-7036-8173-9f72-e255e0174a2f.snapshot
has already existed in destination cluster
2021-02-26 10:07:01,233 INFO [main]
migration.CubeMigrationCrossClusterCLI:590 : Item
/table_snapshot/DEFAULT.KYLIN_COUNTRY/e8d8fe45-06a6-b9cc-354f-b76e4e8007a2.snapshot
has already existed in destination cluster
2021-02-26 10:07:01,234 INFO [main]
migration.CubeMigrationCrossClusterCLI:510 : add segment
kylin_sales_cube[20120101000000_20120701000000] to migration list
2021-02-26 10:07:01,307 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
migration.CubeMigrationCrossClusterCLI:695 : start to copy hdfs directory
from
hdfs://ip-172-31-81-255.ec2.internal:8020/user/hbase/data/default/KYLIN_5CKFDB9L12
to
hdfs://ip-172-31-91-198.ec2.internal:8020/kylin/kylin_metadata/migration/hbase/data/default/KYLIN_5CKFDB9L12
2021-02-26 10:07:01,307 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
migration.CubeMigrationCrossClusterCLI:695 : start to copy hdfs directory
from
hdfs://ip-172-31-81-255.ec2.internal:8020/kylin/kylin_metadata/kylin-4f8a3d06-8bd0-ec10-81ef-55c4a06ac66a
to
hdfs://ip-172-31-91-198.ec2.internal:8020/kylin/kylin_metadata/kylin-4f8a3d06-8bd0-ec10-81ef-55c4a06ac66a
2021-02-26 10:07:01,495 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-81-255.ec2.internal/172.31.81.255:8032
2021-02-26 10:07:01,495 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-81-255.ec2.internal/172.31.81.255:8032
2021-02-26 10:07:01,689 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-81-255.ec2.internal/172.31.81.255:8032
2021-02-26 10:07:01,695 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-81-255.ec2.internal/172.31.81.255:8032
2021-02-26 10:07:01,915 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
tools.SimpleCopyListing:589 : Paths (files+dirs) cnt = 71; dirCnt = 15
2021-02-26 10:07:01,916 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
tools.SimpleCopyListing:384 : Build file listing completed.
2021-02-26 10:07:01,915 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
tools.SimpleCopyListing:589 : Paths (files+dirs) cnt = 12; dirCnt = 6
2021-02-26 10:07:01,916 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
tools.SimpleCopyListing:384 : Build file listing completed.
2021-02-26 10:07:01,917 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
Configuration.deprecation:1285 : io.sort.mb is deprecated. Instead, use
mapreduce.task.io.sort.mb
2021-02-26 10:07:01,917 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
Configuration.deprecation:1285 : io.sort.factor is deprecated. Instead, use
mapreduce.task.io.sort.factor
2021-02-26 10:07:02,397 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
tools.DistCp:94 : Number of paths in the copy list: 12
2021-02-26 10:07:02,402 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
tools.DistCp:94 : Number of paths in the copy list: 71
2021-02-26 10:07:02,427 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
tools.DistCp:94 : Number of paths in the copy list: 12
2021-02-26 10:07:02,432 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
tools.DistCp:94 : Number of paths in the copy list: 71
2021-02-26 10:07:02,466 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-81-255.ec2.internal/172.31.81.255:8032
2021-02-26 10:07:02,466 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-81-255.ec2.internal/172.31.81.255:8032
2021-02-26 10:07:02,514 WARN [Cube-kylin_sales_cube-data-migration-pool-0]
mapreduce.JobResourceUploader:64 : Hadoop command-line option parsing not
performed. Implement the Tool interface and execute your application with
ToolRunner to remedy this.
2021-02-26 10:07:02,514 WARN [Cube-kylin_sales_cube-data-migration-pool-1]
mapreduce.JobResourceUploader:64 : Hadoop command-line option parsing not
performed. Implement the Tool interface and execute your application with
ToolRunner to remedy this.
2021-02-26 10:07:02,637 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
mapreduce.JobSubmitter:200 : number of splits:4
2021-02-26 10:07:02,644 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
mapreduce.JobSubmitter:200 : number of splits:26
2021-02-26 10:07:02,772 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
mapreduce.JobSubmitter:289 : Submitting tokens for job:
job_1614300651018_0027
2021-02-26 10:07:02,789 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
mapreduce.JobSubmitter:289 : Submitting tokens for job:
job_1614300651018_0026
2021-02-26 10:07:02,906 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
mapreduce.JobSubmitter:251 : Cleaning up the staging area
/tmp/hadoop-yarn/staging/hadoop/.staging/job_1614300651018_0027
2021-02-26 10:07:02,906 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
mapreduce.JobSubmitter:251 : Cleaning up the staging area
/tmp/hadoop-yarn/staging/hadoop/.staging/job_1614300651018_0026
2021-02-26 10:07:02,912 ERROR [Cube-kylin_sales_cube-data-migration-pool-1]
migration.CubeMigrationCrossClusterCLI:680 : fail to migrate htable
KYLIN_5CKFDB9L12 due to {}
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:210)
at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:259)
at
org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:467)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:296)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:242)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:193)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:155)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHDFSPath(CubeMigrationCrossClusterCLI.java:701)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHFileByDistCp(CubeMigrationCrossClusterCLI.java:690)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHTable(CubeMigrationCrossClusterCLI.java:662)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHTable(CubeMigrationCrossClusterCLI.java:614)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.access$200(CubeMigrationCrossClusterCLI.java:88)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI$3.doRun(CubeMigrationCrossClusterCLI.java:505)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI$MyRunnable.run(CubeMigrationCrossClusterCLI.java:742)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-02-26 10:07:02,914 ERROR [main]
migration.CubeMigrationCrossClusterCLI:431 : java.lang.NullPointerException
2021-02-26 10:07:02,915 ERROR [main]
migration.CubeMigrationCrossClusterCLI:332 : fail to migrate cube
CUBE[name=kylin_sales_cube] due to
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.NullPointerException
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.migrateCube(CubeMigrationCrossClusterCLI.java:432)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.execute(CubeMigrationCrossClusterCLI.java:329)
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.main(CubeMigrationCrossClusterCLI.java:755)
Caused by: java.util.concurrent.ExecutionException:
java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.migrateCube(CubeMigrationCrossClusterCLI.java:426)
... 3 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:210)
at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:259)
at
org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:467)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:296)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:242)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:193)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:155)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHDFSPath(CubeMigrationCrossClusterCLI.java:701)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHDFSJobInfo(CubeMigrationCrossClusterCLI.java:602)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.access$100(CubeMigrationCrossClusterCLI.java:88)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI$2.doRun(CubeMigrationCrossClusterCLI.java:497)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI$MyRunnable.run(CubeMigrationCrossClusterCLI.java:742)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-02-26 10:07:02,915 INFO [main] migration.DstClusterUtil:363 : update
meta cache for org.apache.kylin.common.restclient.RestClient@7bab5898
2021-02-26 10:07:02,920 WARN [main]
migration.CubeMigrationCrossClusterCLI:361 : Failed to migrate cubes
[CUBE[name=kylin_sales_cube]] and need to check the detailed reason and
retry again!!!
2021-02-26 10:07:02,924 INFO [close-hbase-conn] hbase.HBaseConnection:137 :
Closing HBase connections...
2021-02-26 10:07:02,924 INFO [close-hbase-conn]
client.ConnectionManager$HConnectionImplementation:2259 : Closing master
protocol: MasterService
2021-02-26 10:07:02,929 INFO [close-hbase-conn]
client.ConnectionManager$HConnectionImplementation:1776 : Closing zookeeper
sessionid=0x1000002a32a0021
2021-02-26 10:07:02,930 INFO [close-hbase-conn] zookeeper.ZooKeeper:693 :
Session: 0x1000002a32a0021 closed
2021-02-26 10:07:02,930 INFO [main-EventThread] zookeeper.ClientCnxn:522 :
EventThread shut down for session: 0x1000002a32a0021
[hadoop@ip-172-31-81-255 apache-kylin-3.1.0-bin-hbase1x]$
Please let me know if I am missing something
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: Cube migration across hadoop clusters
Posted by Zhichao Zhang <44...@qq.com>.
Hi, please add this parameter '-schemaOnly false' in CMD '${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI'.
schemaOnly parameter specifies whether only migrate cube related schema, the default value is true, when set to false, it will migrate data too.
Best regards,
Zhichao Zhang
Re: Cube migration across hadoop clusters
Posted by xatax <ro...@gmail.com>.
hi Yaqian,
thanks for the response. I am using this method from the documentation: "II.
Migrate across two Hadoop clusters" using Kylin version 3.1.0 on EMR 5.29 on
both source and destination clusters.
I tried in 2 ways:
(1) Migrate from one kylin instance on emr 5.29 to another kylin instance on
emr 5.29 with the Hbase S3 location same on both.
(2) Migrate from one kylin instance on emr 5.29 to another kylin instance on
emr 5.29 with Hbase s3 location different.
In both ways the cube metadata/definition was migrated but not the cube
data.
Here is the log for (1):
[hadoop@ip-172-31-90-136 apache-kylin-3.1.0-bin-hbase1x]$ ./bin/kylin.sh
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI -kylinUriSrc
ADMIN:KYLIN@ip-172-31-90-136.ec2.internal:7070 -kylinUriDst
ADMIN:KYLIN@ip-172-31-81-180.ec2.internal:7070 -cube kylin_sales_cube
-execute true -overwrite true
Using cached dependency...
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support
was removed in 8.0
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x/tool/kylin-tool-3.1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Running org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI
-kylinUriSrc ADMIN:KYLIN@ip-172-31-90-136.ec2.internal:7070 -kylinUriDst
ADMIN:KYLIN@ip-172-31-81-180.ec2.internal:7070 -cube kylin_sales_cube
-execute true -overwrite true
2021-02-25 09:36:50,551 INFO [main] common.KylinConfig:118 : Loading
kylin-defaults.properties from
file:/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x/tool/kylin-tool-3.1.0.jar!/kylin-defaults.properties
2021-02-25 09:36:50,576 DEBUG [main] common.KylinConfig:346 : KYLIN_CONF
property was not set, will seek KYLIN_HOME env variable
2021-02-25 09:36:50,579 INFO [main] common.KylinConfig:352 : Use
KYLIN_HOME=/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x
2021-02-25 09:36:50,582 INFO [main] common.KylinConfig:153 : Initialized a
new KylinConfig from getInstanceFromEnv : 245765246
2021-02-25 09:36:55,215 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x5d3b58ca connecting to ZooKeeper
ensemble=ip-172-31-90-136.ec2.internal:2181
2021-02-25 09:36:55,224 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:zookeeper.version=3.4.14-5f788d5e8f90539ee331702c753fa250727128f4,
built on 12/14/2019 00:35 GMT
2021-02-25 09:36:55,224 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:host.name=ip-172-31-90-136.ec2.internal
2021-02-25 09:36:55,225 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.version=1.8.0_272
2021-02-25 09:36:55,225 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.vendor=Red Hat, Inc.
2021-02-25 09:36:55,225 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.56.amzn1.x86_64/jre
2021-02-25 09:36:55,225 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.class.path=/*removed*/
2021-02-25 09:36:55,226 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.library.path=:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native
2021-02-25 09:36:55,227 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.io.tmpdir=/tmp
2021-02-25 09:36:55,227 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.compiler=<NA>
2021-02-25 09:36:55,227 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.name=Linux
2021-02-25 09:36:55,228 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.arch=amd64
2021-02-25 09:36:55,229 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.version=4.14.154-99.181.amzn1.x86_64
2021-02-25 09:36:55,229 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.name=hadoop
2021-02-25 09:36:55,229 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.home=/home/hadoop
2021-02-25 09:36:55,229 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.dir=/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x
2021-02-25 09:36:55,230 INFO [main] zookeeper.ZooKeeper:442 : Initiating
client connection, connectString=ip-172-31-90-136.ec2.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7e91ed74
2021-02-25 09:36:55,289 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:1025 : Opening socket connection to server
ip-172-31-90-136.ec2.internal/172.31.90.136:2181. Will not attempt to
authenticate using SASL (unknown error)
2021-02-25 09:36:55,303 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:879 : Socket connection established to
ip-172-31-90-136.ec2.internal/172.31.90.136:2181, initiating session
2021-02-25 09:36:55,309 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-90-136.ec2.internal/172.31.90.136:2181, sessionid =
0x10000029296003a, negotiated timeout = 40000
2021-02-25 09:36:55,787 DEBUG [main] hbase.HBaseConnection:316 : HTable
'kylin_metadata' already exists
2021-02-25 09:36:56,135 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.TableMetadataManager
2021-02-25 09:36:56,142 INFO [main] persistence.ResourceStore:90 : Using
metadata url kylin_metadata@hbase for resource store
2021-02-25 09:36:56,207 INFO [main] hbase.HBaseConnection:267 : connection
is null or closed, creating a new one
2021-02-25 09:36:56,207 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x3c25cfe1 connecting to ZooKeeper
ensemble=ip-172-31-90-136.ec2.internal:2181
2021-02-25 09:36:56,208 INFO [main] zookeeper.ZooKeeper:442 : Initiating
client connection, connectString=ip-172-31-90-136.ec2.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1d3c112a
2021-02-25 09:36:56,210 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:1025 : Opening socket connection to server
ip-172-31-90-136.ec2.internal/172.31.90.136:2181. Will not attempt to
authenticate using SASL (unknown error)
2021-02-25 09:36:56,211 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:879 : Socket connection established to
ip-172-31-90-136.ec2.internal/172.31.90.136:2181, initiating session
2021-02-25 09:36:56,213 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-90-136.ec2.internal/172.31.90.136:2181, sessionid =
0x10000029296003b, negotiated timeout = 40000
2021-02-25 09:36:56,219 DEBUG [main] hbase.HBaseConnection:316 : HTable
'kylin_metadata' already exists
2021-02-25 09:36:56,478 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading TableDesc from kylin_metadata(key='/table')@kylin_metadata@hbase
2021-02-25 09:36:56,546 INFO [main] measure.MeasureTypeFactory:121 :
Checking custom measure types from kylin config
2021-02-25 09:36:56,547 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(hllc), class
org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
2021-02-25 09:36:56,551 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(bitmap), class
org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
2021-02-25 09:36:56,552 INFO [main] measure.MeasureTypeFactory:150 :
registering TOP_N(topn), class
org.apache.kylin.measure.topn.TopNMeasureType$Factory
2021-02-25 09:36:56,554 INFO [main] measure.MeasureTypeFactory:150 :
registering RAW(raw), class
org.apache.kylin.measure.raw.RawMeasureType$Factory
2021-02-25 09:36:56,554 INFO [main] measure.MeasureTypeFactory:150 :
registering EXTENDED_COLUMN(extendedcolumn), class
org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
2021-02-25 09:36:56,555 INFO [main] measure.MeasureTypeFactory:150 :
registering PERCENTILE_APPROX(percentile), class
org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
2021-02-25 09:36:56,556 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(dim_dc), class
org.apache.kylin.measure.dim.DimCountDistinctMeasureType$Factory
2021-02-25 09:36:56,557 INFO [main] measure.MeasureTypeFactory:150 :
registering STDDEV_SUM(stddev_sum), class
org.apache.kylin.measure.stddev.StdDevSumMeasureType$Factory
2021-02-25 09:36:56,560 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(bitmap_map), class
org.apache.kylin.measure.map.bitmap.BitmapMapMeasureType$Factory
2021-02-25 09:36:56,563 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
6 TableDesc(s) out of 6 resource with 0 errors
2021-02-25 09:36:56,564 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2021-02-25 09:36:56,565 DEBUG [main] cachesync.Broadcaster:102 : 1 nodes in
the cluster: [localhost:7070]
2021-02-25 09:36:56,566 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading TableExtDesc from
kylin_metadata(key='/table_exd')@kylin_metadata@hbase
2021-02-25 09:36:56,568 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 TableExtDesc(s) out of 0 resource with 0 errors
2021-02-25 09:36:56,568 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading ExternalFilterDesc from
kylin_metadata(key='/ext_filter')@kylin_metadata@hbase
2021-02-25 09:36:56,570 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 ExternalFilterDesc(s) out of 0 resource with 0 errors
2021-02-25 09:36:56,572 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.model.DataModelManager
2021-02-25 09:36:56,573 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading DataModelDesc from
kylin_metadata(key='/model_desc')@kylin_metadata@hbase
2021-02-25 09:36:56,593 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.project.ProjectManager
2021-02-25 09:36:56,593 INFO [main] project.ProjectManager:85 :
Initializing ProjectManager with metadata url kylin_metadata@hbase
2021-02-25 09:36:56,594 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading ProjectInstance from
kylin_metadata(key='/project')@kylin_metadata@hbase
2021-02-25 09:36:56,602 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
1 ProjectInstance(s) out of 1 resource with 0 errors
2021-02-25 09:36:56,609 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
2 DataModelDesc(s) out of 2 resource with 0 errors
2021-02-25 09:36:56,611 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.storage.hybrid.HybridManager
2021-02-25 09:36:56,611 INFO [main] hybrid.HybridManager:71 : Initializing
HybridManager with config kylin_metadata@hbase
2021-02-25 09:36:56,611 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading HybridInstance from
kylin_metadata(key='/hybrid')@kylin_metadata@hbase
2021-02-25 09:36:56,613 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 HybridInstance(s) out of 0 resource with 0 errors
2021-02-25 09:36:56,616 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.cube.CubeManager
2021-02-25 09:36:56,623 INFO [main] cube.CubeManager:136 : Initializing
CubeManager with config kylin_metadata@hbase
2021-02-25 09:36:56,623 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading CubeInstance from kylin_metadata(key='/cube')@kylin_metadata@hbase
2021-02-25 09:36:56,703 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.cube.CubeDescManager
2021-02-25 09:36:56,704 INFO [main] cube.CubeDescManager:91 : Initializing
CubeDescManager with config kylin_metadata@hbase
2021-02-25 09:36:56,704 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading CubeDesc from
kylin_metadata(key='/cube_desc')@kylin_metadata@hbase
2021-02-25 09:36:56,819 INFO [main] model.RowKeyColDesc:99 : Implicitly
convert encoding to time
2021-02-25 09:36:56,819 INFO [main] model.RowKeyColDesc:99 : Implicitly
convert encoding to time
2021-02-25 09:36:56,819 INFO [main] model.RowKeyColDesc:95 : Implicitly
convert encoding to date
2021-02-25 09:36:56,820 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
2 CubeDesc(s) out of 2 resource with 0 errors
2021-02-25 09:36:56,820 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
2 CubeInstance(s) out of 2 resource with 0 errors
2021-02-25 09:36:56,821 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class
org.apache.kylin.metadata.realization.RealizationRegistry
2021-02-25 09:36:56,821 INFO [main] realization.RealizationRegistry:54 :
Initializing RealizationRegistry with metadata url kylin_metadata@hbase
2021-02-25 09:36:56,821 INFO [main] realization.RealizationRegistry:81 :
RealizationRegistry is {CUBE=org.apache.kylin.cube.CubeManager@1a1f79ce,
HYBRID=org.apache.kylin.storage.hybrid.HybridManager@6aa7e176}
2021-02-25 09:36:56,822 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.dict.DictionaryManager
2021-02-25 09:36:56,823 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.dict.lookup.SnapshotManager
2021-02-25 09:36:57,143 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x2589d787 connecting to ZooKeeper
ensemble=ip-172-31-81-180.ec2.internal:2181
2021-02-25 09:36:57,143 INFO [main] zookeeper.ZooKeeper:442 : Initiating
client connection, connectString=ip-172-31-81-180.ec2.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1788cb61
2021-02-25 09:36:57,153 INFO
[main-SendThread(ip-172-31-81-180.ec2.internal:2181)]
zookeeper.ClientCnxn:1025 : Opening socket connection to server
ip-172-31-81-180.ec2.internal/172.31.81.180:2181. Will not attempt to
authenticate using SASL (unknown error)
2021-02-25 09:36:57,154 INFO
[main-SendThread(ip-172-31-81-180.ec2.internal:2181)]
zookeeper.ClientCnxn:879 : Socket connection established to
ip-172-31-81-180.ec2.internal/172.31.81.180:2181, initiating session
2021-02-25 09:36:57,156 INFO
[main-SendThread(ip-172-31-81-180.ec2.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-81-180.ec2.internal/172.31.81.180:2181, sessionid =
0x100000279680020, negotiated timeout = 40000
2021-02-25 09:36:57,170 DEBUG [main] hbase.HBaseConnection:316 : HTable
'kylin_metadata' already exists
2021-02-25 09:36:57,178 INFO [main]
migration.CubeMigrationCrossClusterCLI:327 : start to migrate cube
CUBE[name=kylin_sales_cube]
2021-02-25 09:36:59,671 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_CAL_DT.json
2021-02-25 09:36:59,672 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_CAL_DT', columns= /*removed*/
2021-02-25 09:36:59,676 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_CATEGORY_GROUPINGS.json
2021-02-25 09:36:59,676 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_CATEGORY_GROUPINGS', columns= /*removed*/
2021-02-25 09:36:59,678 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_ACCOUNT.json
2021-02-25 09:36:59,679 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_ACCOUNT', columns=/*removed*/
2021-02-25 09:36:59,681 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_COUNTRY.json
2021-02-25 09:36:59,681 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_COUNTRY', columns=/*removed*/
2021-02-25 09:36:59,683 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_SALES.json
2021-02-25 09:36:59,683 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_SALES', columns=/*removed*/
2021-02-25 09:36:59,706 INFO [main] migration.DstClusterUtil:249 : saved
resource /model_desc/kylin_sales_model.json
2021-02-25 09:36:59,706 INFO [main] migration.DstClusterUtil:170 : saved
model DataModelDesc [name=kylin_sales_model]
2021-02-25 09:36:59,726 INFO [main] migration.DstClusterUtil:249 : saved
resource /cube_desc/kylin_sales_cube.json
2021-02-25 09:36:59,726 INFO [main] migration.DstClusterUtil:178 : saved
cube desc CubeDesc [name=kylin_sales_cube]
2021-02-25 09:36:59,731 INFO [main] migration.DstClusterUtil:249 : saved
resource /cube/kylin_sales_cube.json
2021-02-25 09:36:59,731 INFO [main] migration.DstClusterUtil:185 : saved
cube instance CUBE[name=kylin_sales_cube]
2021-02-25 09:36:59,739 INFO [main]
migration.CubeMigrationCrossClusterCLI:330 : finish migrating cube
CUBE[name=kylin_sales_cube]
2021-02-25 09:36:59,745 INFO [main] migration.DstClusterUtil:249 : saved
resource /project/learn_kylin.json
2021-02-25 09:36:59,745 INFO [main] migration.DstClusterUtil:146 : saved
project ProjectDesc [name=learn_kylin]
2021-02-25 09:36:59,745 INFO [main] migration.DstClusterUtil:363 : update
meta cache for org.apache.kylin.common.restclient.RestClient@58d4238e
2021-02-25 09:36:59,749 INFO [main]
migration.CubeMigrationCrossClusterCLI:359 : Migration for cubes
[CUBE[name=kylin_sales_cube]], hyrbids [] all succeed
2021-02-25 09:36:59,751 INFO [close-hbase-conn] hbase.HBaseConnection:137 :
Closing HBase connections...
2021-02-25 09:36:59,751 INFO [close-hbase-conn]
client.ConnectionManager$HConnectionImplementation:2259 : Closing master
protocol: MasterService
2021-02-25 09:36:59,753 INFO [close-hbase-conn]
client.ConnectionManager$HConnectionImplementation:1776 : Closing zookeeper
sessionid=0x10000029296003b
2021-02-25 09:36:59,755 INFO [close-hbase-conn] zookeeper.ZooKeeper:693 :
Session: 0x10000029296003b closed
2021-02-25 09:36:59,755 INFO [main-EventThread] zookeeper.ClientCnxn:522 :
EventThread shut down for session: 0x10000029296003b
Here is the log for (2):
[hadoop@ip-172-31-90-136 ~]$ cd $KYLIN_HOME
[hadoop@ip-172-31-90-136 apache-kylin-3.1.0-bin-hbase1x]$ ./bin/kylin.sh
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI -kylinUriSrc
ADMIN:KYLIN@ip-172-31-90-136.ec2.internal:7070 -kylinUriDst
ADMIN:KYLIN@ip-172-31-85-14.ec2.internal:7070 -cube kylin_sales_cube
-execute true -overwrite true
Using cached dependency...
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support
was removed in 8.0
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x/tool/kylin-tool-3.1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Running org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI
-kylinUriSrc ADMIN:KYLIN@ip-172-31-90-136.ec2.internal:7070 -kylinUriDst
ADMIN:KYLIN@ip-172-31-85-14.ec2.internal:7070 -cube kylin_sales_cube
-execute true -overwrite true
2021-02-25 09:44:09,900 INFO [main] common.KylinConfig:118 : Loading
kylin-defaults.properties from
file:/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x/tool/kylin-tool-3.1.0.jar!/kylin-defaults.properties
2021-02-25 09:44:09,935 DEBUG [main] common.KylinConfig:346 : KYLIN_CONF
property was not set, will seek KYLIN_HOME env variable
2021-02-25 09:44:09,937 INFO [main] common.KylinConfig:352 : Use
KYLIN_HOME=/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x
2021-02-25 09:44:09,940 INFO [main] common.KylinConfig:153 : Initialized a
new KylinConfig from getInstanceFromEnv : 245765246
2021-02-25 09:44:14,486 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x5d3b58ca connecting to ZooKeeper
ensemble=ip-172-31-90-136.ec2.internal:2181
2021-02-25 09:44:14,492 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:zookeeper.version=3.4.14-5f788d5e8f90539ee331702c753fa250727128f4,
built on 12/14/2019 00:35 GMT
2021-02-25 09:44:14,492 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:host.name=ip-172-31-90-136.ec2.internal
2021-02-25 09:44:14,493 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.version=1.8.0_272
2021-02-25 09:44:14,493 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.vendor=Red Hat, Inc.
2021-02-25 09:44:14,493 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.56.amzn1.x86_64/jre
2021-02-25 09:44:14,493 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.class.path=/*REMOVED*/
2021-02-25 09:44:14,494 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.library.path=:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native
2021-02-25 09:44:14,494 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.io.tmpdir=/tmp
2021-02-25 09:44:14,494 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.compiler=<NA>
2021-02-25 09:44:14,495 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.name=Linux
2021-02-25 09:44:14,495 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.arch=amd64
2021-02-25 09:44:14,496 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.version=4.14.154-99.181.amzn1.x86_64
2021-02-25 09:44:14,496 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.name=hadoop
2021-02-25 09:44:14,496 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.home=/home/hadoop
2021-02-25 09:44:14,496 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.dir=/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x
2021-02-25 09:44:14,497 INFO [main] zookeeper.ZooKeeper:442 : Initiating
client connection, connectString=ip-172-31-90-136.ec2.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7e91ed74
2021-02-25 09:44:14,564 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:1025 : Opening socket connection to server
ip-172-31-90-136.ec2.internal/172.31.90.136:2181. Will not attempt to
authenticate using SASL (unknown error)
2021-02-25 09:44:14,569 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:879 : Socket connection established to
ip-172-31-90-136.ec2.internal/172.31.90.136:2181, initiating session
2021-02-25 09:44:14,574 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-90-136.ec2.internal/172.31.90.136:2181, sessionid =
0x10000029296003c, negotiated timeout = 40000
2021-02-25 09:44:15,012 DEBUG [main] hbase.HBaseConnection:316 : HTable
'kylin_metadata' already exists
2021-02-25 09:44:15,356 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.TableMetadataManager
2021-02-25 09:44:15,364 INFO [main] persistence.ResourceStore:90 : Using
metadata url kylin_metadata@hbase for resource store
2021-02-25 09:44:15,426 INFO [main] hbase.HBaseConnection:267 : connection
is null or closed, creating a new one
2021-02-25 09:44:15,426 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x3c25cfe1 connecting to ZooKeeper
ensemble=ip-172-31-90-136.ec2.internal:2181
2021-02-25 09:44:15,427 INFO [main] zookeeper.ZooKeeper:442 : Initiating
client connection, connectString=ip-172-31-90-136.ec2.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1d3c112a
2021-02-25 09:44:15,428 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:1025 : Opening socket connection to server
ip-172-31-90-136.ec2.internal/172.31.90.136:2181. Will not attempt to
authenticate using SASL (unknown error)
2021-02-25 09:44:15,428 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:879 : Socket connection established to
ip-172-31-90-136.ec2.internal/172.31.90.136:2181, initiating session
2021-02-25 09:44:15,430 INFO
[main-SendThread(ip-172-31-90-136.ec2.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-90-136.ec2.internal/172.31.90.136:2181, sessionid =
0x10000029296003d, negotiated timeout = 40000
2021-02-25 09:44:15,436 DEBUG [main] hbase.HBaseConnection:316 : HTable
'kylin_metadata' already exists
2021-02-25 09:44:15,552 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading TableDesc from kylin_metadata(key='/table')@kylin_metadata@hbase
2021-02-25 09:44:15,626 INFO [main] measure.MeasureTypeFactory:121 :
Checking custom measure types from kylin config
2021-02-25 09:44:15,627 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(hllc), class
org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
2021-02-25 09:44:15,630 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(bitmap), class
org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
2021-02-25 09:44:15,632 INFO [main] measure.MeasureTypeFactory:150 :
registering TOP_N(topn), class
org.apache.kylin.measure.topn.TopNMeasureType$Factory
2021-02-25 09:44:15,633 INFO [main] measure.MeasureTypeFactory:150 :
registering RAW(raw), class
org.apache.kylin.measure.raw.RawMeasureType$Factory
2021-02-25 09:44:15,634 INFO [main] measure.MeasureTypeFactory:150 :
registering EXTENDED_COLUMN(extendedcolumn), class
org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
2021-02-25 09:44:15,635 INFO [main] measure.MeasureTypeFactory:150 :
registering PERCENTILE_APPROX(percentile), class
org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
2021-02-25 09:44:15,637 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(dim_dc), class
org.apache.kylin.measure.dim.DimCountDistinctMeasureType$Factory
2021-02-25 09:44:15,637 INFO [main] measure.MeasureTypeFactory:150 :
registering STDDEV_SUM(stddev_sum), class
org.apache.kylin.measure.stddev.StdDevSumMeasureType$Factory
2021-02-25 09:44:15,641 INFO [main] measure.MeasureTypeFactory:150 :
registering COUNT_DISTINCT(bitmap_map), class
org.apache.kylin.measure.map.bitmap.BitmapMapMeasureType$Factory
2021-02-25 09:44:15,644 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
6 TableDesc(s) out of 6 resource with 0 errors
2021-02-25 09:44:15,645 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2021-02-25 09:44:15,646 DEBUG [main] cachesync.Broadcaster:102 : 1 nodes in
the cluster: [localhost:7070]
2021-02-25 09:44:15,647 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading TableExtDesc from
kylin_metadata(key='/table_exd')@kylin_metadata@hbase
2021-02-25 09:44:15,649 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 TableExtDesc(s) out of 0 resource with 0 errors
2021-02-25 09:44:15,650 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading ExternalFilterDesc from
kylin_metadata(key='/ext_filter')@kylin_metadata@hbase
2021-02-25 09:44:15,651 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 ExternalFilterDesc(s) out of 0 resource with 0 errors
2021-02-25 09:44:15,653 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.model.DataModelManager
2021-02-25 09:44:15,654 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading DataModelDesc from
kylin_metadata(key='/model_desc')@kylin_metadata@hbase
2021-02-25 09:44:15,676 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.metadata.project.ProjectManager
2021-02-25 09:44:15,677 INFO [main] project.ProjectManager:85 :
Initializing ProjectManager with metadata url kylin_metadata@hbase
2021-02-25 09:44:15,678 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading ProjectInstance from
kylin_metadata(key='/project')@kylin_metadata@hbase
2021-02-25 09:44:15,687 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
1 ProjectInstance(s) out of 1 resource with 0 errors
2021-02-25 09:44:15,693 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
2 DataModelDesc(s) out of 2 resource with 0 errors
2021-02-25 09:44:15,694 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.storage.hybrid.HybridManager
2021-02-25 09:44:15,695 INFO [main] hybrid.HybridManager:71 : Initializing
HybridManager with config kylin_metadata@hbase
2021-02-25 09:44:15,695 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading HybridInstance from
kylin_metadata(key='/hybrid')@kylin_metadata@hbase
2021-02-25 09:44:15,697 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
0 HybridInstance(s) out of 0 resource with 0 errors
2021-02-25 09:44:15,700 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.cube.CubeManager
2021-02-25 09:44:15,707 INFO [main] cube.CubeManager:136 : Initializing
CubeManager with config kylin_metadata@hbase
2021-02-25 09:44:15,708 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading CubeInstance from kylin_metadata(key='/cube')@kylin_metadata@hbase
2021-02-25 09:44:15,783 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.cube.CubeDescManager
2021-02-25 09:44:15,783 INFO [main] cube.CubeDescManager:91 : Initializing
CubeDescManager with config kylin_metadata@hbase
2021-02-25 09:44:15,783 DEBUG [main] cachesync.CachedCrudAssist:122 :
Reloading CubeDesc from
kylin_metadata(key='/cube_desc')@kylin_metadata@hbase
2021-02-25 09:44:15,873 INFO [main] model.RowKeyColDesc:99 : Implicitly
convert encoding to time
2021-02-25 09:44:15,873 INFO [main] model.RowKeyColDesc:99 : Implicitly
convert encoding to time
2021-02-25 09:44:15,873 INFO [main] model.RowKeyColDesc:95 : Implicitly
convert encoding to date
2021-02-25 09:44:15,873 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
2 CubeDesc(s) out of 2 resource with 0 errors
2021-02-25 09:44:15,874 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded
2 CubeInstance(s) out of 2 resource with 0 errors
2021-02-25 09:44:15,874 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class
org.apache.kylin.metadata.realization.RealizationRegistry
2021-02-25 09:44:15,875 INFO [main] realization.RealizationRegistry:54 :
Initializing RealizationRegistry with metadata url kylin_metadata@hbase
2021-02-25 09:44:15,875 INFO [main] realization.RealizationRegistry:81 :
RealizationRegistry is {CUBE=org.apache.kylin.cube.CubeManager@1a1f79ce,
HYBRID=org.apache.kylin.storage.hybrid.HybridManager@6aa7e176}
2021-02-25 09:44:15,876 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.dict.DictionaryManager
2021-02-25 09:44:15,876 INFO [main] common.KylinConfig:476 : Creating new
manager instance of class org.apache.kylin.dict.lookup.SnapshotManager
2021-02-25 09:44:16,138 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x2589d787 connecting to ZooKeeper
ensemble=ip-172-31-85-14.ec2.internal:2181
2021-02-25 09:44:16,138 INFO [main] zookeeper.ZooKeeper:442 : Initiating
client connection, connectString=ip-172-31-85-14.ec2.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1788cb61
2021-02-25 09:44:16,140 INFO
[main-SendThread(ip-172-31-85-14.ec2.internal:2181)]
zookeeper.ClientCnxn:1025 : Opening socket connection to server
ip-172-31-85-14.ec2.internal/172.31.85.14:2181. Will not attempt to
authenticate using SASL (unknown error)
2021-02-25 09:44:16,141 INFO
[main-SendThread(ip-172-31-85-14.ec2.internal:2181)]
zookeeper.ClientCnxn:879 : Socket connection established to
ip-172-31-85-14.ec2.internal/172.31.85.14:2181, initiating session
2021-02-25 09:44:16,143 INFO
[main-SendThread(ip-172-31-85-14.ec2.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-85-14.ec2.internal/172.31.85.14:2181, sessionid =
0x10000028a4b001f, negotiated timeout = 40000
2021-02-25 09:44:16,150 DEBUG [main] hbase.HBaseConnection:316 : HTable
'kylin_metadata' already exists
2021-02-25 09:44:16,156 INFO [main]
migration.CubeMigrationCrossClusterCLI:327 : start to migrate cube
CUBE[name=kylin_sales_cube]
2021-02-25 09:44:18,950 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_CAL_DT.json
2021-02-25 09:44:18,951 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_CAL_DT', columns=/*REMOVED*/
2021-02-25 09:44:18,955 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_CATEGORY_GROUPINGS.json
2021-02-25 09:44:18,955 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_CATEGORY_GROUPINGS', columns=/*REMOVED*/
2021-02-25 09:44:18,959 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_ACCOUNT.json
2021-02-25 09:44:18,959 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_ACCOUNT', columns=/*REMOVED*/
2021-02-25 09:44:18,979 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_COUNTRY.json
2021-02-25 09:44:18,979 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_COUNTRY', columns=/*REMOVED*/
2021-02-25 09:44:18,982 INFO [main] migration.DstClusterUtil:249 : saved
resource /table/DEFAULT.KYLIN_SALES.json
2021-02-25 09:44:18,982 INFO [main] migration.DstClusterUtil:162 : saved
table TableDesc{name='KYLIN_SALES', columns=/*REMOVED*/
2021-02-25 09:44:18,985 INFO [main] migration.DstClusterUtil:249 : saved
resource /model_desc/kylin_sales_model.json
2021-02-25 09:44:18,985 INFO [main] migration.DstClusterUtil:170 : saved
model DataModelDesc [name=kylin_sales_model]
2021-02-25 09:44:19,006 INFO [main] migration.DstClusterUtil:249 : saved
resource /cube_desc/kylin_sales_cube.json
2021-02-25 09:44:19,007 INFO [main] migration.DstClusterUtil:178 : saved
cube desc CubeDesc [name=kylin_sales_cube]
2021-02-25 09:44:19,012 INFO [main] migration.DstClusterUtil:249 : saved
resource /cube/kylin_sales_cube.json
2021-02-25 09:44:19,012 INFO [main] migration.DstClusterUtil:185 : saved
cube instance CUBE[name=kylin_sales_cube]
2021-02-25 09:44:19,020 INFO [main]
migration.CubeMigrationCrossClusterCLI:330 : finish migrating cube
CUBE[name=kylin_sales_cube]
2021-02-25 09:44:19,026 INFO [main] migration.DstClusterUtil:249 : saved
resource /project/learn_kylin.json
2021-02-25 09:44:19,026 INFO [main] migration.DstClusterUtil:146 : saved
project ProjectDesc [name=learn_kylin]
2021-02-25 09:44:19,026 INFO [main] migration.DstClusterUtil:363 : update
meta cache for org.apache.kylin.common.restclient.RestClient@58d4238e
2021-02-25 09:44:19,030 INFO [main]
migration.CubeMigrationCrossClusterCLI:359 : Migration for cubes
[CUBE[name=kylin_sales_cube]], hyrbids [] all succeed
2021-02-25 09:44:19,032 INFO [close-hbase-conn] hbase.HBaseConnection:137 :
Closing HBase connections...
2021-02-25 09:44:19,034 INFO [close-hbase-conn]
client.ConnectionManager$HConnectionImplementation:2259 : Closing master
protocol: MasterService
2021-02-25 09:44:19,037 INFO [close-hbase-conn]
client.ConnectionManager$HConnectionImplementation:1776 : Closing zookeeper
sessionid=0x10000029296003d
2021-02-25 09:44:19,039 INFO [main-EventThread] zookeeper.ClientCnxn:522 :
EventThread shut down for session: 0x10000029296003d
2021-02-25 09:44:19,039 INFO [close-hbase-conn] zookeeper.ZooKeeper:693 :
Session: 0x10000029296003d closed
[hadoop@ip-172-31-90-136 apache-kylin-3.1.0-bin-hbase1x]$
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: Cube migration across hadoop clusters
Posted by Yaqian Zhang <Ya...@126.com>.
Hi:
There are three ways of cube migration in the document of cube migration. Which one do you use? And what's your kylin version? Is the Hadoop version used by QA cluster and dev cluster consistent? Are there any errors reported during migration?
> 在 2021年2月25日，03:04，xatax <ro...@gmail.com> 写道：
>
> Per: http://kylin.apache.org/docs/tutorial/cube_migration.html
>
> It says: "Support to migrate cube data (segments data on HBase) from QA env
> to PROD env."
>
> I tried the migration but it only migrates the cube metadata without the
> data. The source kylin instance has the cube buiilt and in ready state.
> After I executed the command it said it was successful. But when I checked
> the destination kylin instance it only got the cube definition and not the
> data.
>
> Can you please clarify if migrating the data is possible or not?
>
> Or is the migration of the data a separate activity which involves the hbase
> snapshot import export.
>
> thanks in advance
>
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/