You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/05/14 09:49:57 UTC
[GitHub] [iceberg] dixingxing0 opened a new issue #2198: Hive: got error while joining iceberg table and hive table
dixingxing0 opened a new issue #2198:
URL: https://github.com/apache/iceberg/issues/2198
I got an error while join iceberg table and hive table at same time, most of mappers succeed, but some mappers got an `cannot find field` error:
```java
Caused by: java.lang.RuntimeException: cannot find field log_src from [org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObjectInspector$IcebergRecordStructField@8838736]
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:442)
at org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObjectInspector.getStructFieldRef(IcebergRecordObjectInspector.java:78)
at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:139)
at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:980)
at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:1006)
at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:77)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:498)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:115)
... 22 more
```
Environment:
```java
Iceberg : 0.11
hive : 2.0.1
hadoop : 2.7.2
JDK : 1.8
```
I found some clues:
```java
2021-02-02 15:57:35,036 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/tmp/hive/flink/25369d0b-110e-4eca-8553-f2f3953f0303/hive_2021-02-02_15-56-22_060_2459779186626852210-1/-mr-10005/0/emptyFile:0+0InputFormatClass: org.apache.hadoop.mapred.TextInputFormat
```
The split file is for `hive_table1`(which is an empty table for test), but still the mapper using `org.apache.iceberg.mr.hive.HiveIcebergSerDe`, the table properties is also from `iceberg_table1`.
It seems like a bug of hive, hive using wrong serd to deserialize data, I am not an expert in hive, hope someone could give me some clues .
I've created an hive issue here: [HIVE-24720](https://issues.apache.org/jira/browse/HIVE-24720)
SQL:
```sql
select count(1),count(a.dt),count(b.dt)
from (
select dt,concat_ws('###',wx_source, log_src, dt, hour) as str
from flink_fdm_iceberg.iceberg_table1
where dt='2021-01-29' and hour='16') a
full outer join (
select dt,concat_ws('###',wx_source, log_src, dt, hour) as str
from flink_fdm_iceberg.hive_table1
where dt='2021-01-29' and hour='16') b on a.str=b.str
where a.str is null or b.str is null;
```
The full log by the failed mapper:
```java
2021-02-02 15:57:30,060 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2021-02-02 15:57:30,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2021-02-02 15:57:30,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2021-02-02 15:57:30,131 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2021-02-02 15:57:30,131 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1605975810748_1427, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@4b2bac3f)
2021-02-02 15:57:30,167 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns2, Ident: (HDFS_DELEGATION_TOKEN token 151282235 for flink)
2021-02-02 15:57:30,168 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident: (HDFS_DELEGATION_TOKEN token 151249944 for flink)
2021-02-02 15:57:30,168 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns4, Ident: (HDFS_DELEGATION_TOKEN token 151235141 for flink)
2021-02-02 15:57:30,169 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns3, Ident: (HDFS_DELEGATION_TOKEN token 151183885 for flink)
2021-02-02 15:57:30,169 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns6, Ident: (HDFS_DELEGATION_TOKEN token 168039224 for flink)
2021-02-02 15:57:30,169 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns5, Ident: (HDFS_DELEGATION_TOKEN token 150801000 for flink)
2021-02-02 15:57:30,881 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2021-02-02 15:57:31,032 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data1/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data2/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data4/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data5/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data6/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data7/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data8/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data9/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data10/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data11/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data12/nm-local-dir/usercache/flink/appcache/
application_1605975810748_1427
2021-02-02 15:57:31,937 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2021-02-02 15:57:34,143 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
// !!!!!
2021-02-02 15:57:35,036 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/tmp/hive/flink/25369d0b-110e-4eca-8553-f2f3953f0303/hive_2021-02-02_15-56-22_060_2459779186626852210-1/-mr-10005/0/emptyFile:0+0InputFormatClass: org.apache.hadoop.mapred.TextInputFormat
2021-02-02 15:57:35,198 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH = hdfs://ns6/tmp/hive/flink/25369d0b-110e-4eca-8553-f2f3953f0303/hive_2021-02-02_15-56-22_060_2459779186626852210-1/-mr-10006/33cfad1a-7384-4c27-b2d4-2f894ecb845a/map.xml
2021-02-02 15:57:35,209 WARN [main] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
2021-02-02 15:57:35,221 INFO [main] org.apache.hadoop.hive.ql.exec.SerializationUtilities: Deserializing MapWork using kryo
2021-02-02 15:57:35,384 INFO [main] com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries
2021-02-02 15:57:35,386 INFO [main] com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 5e360bdefd8923280b1a0234b845448050bf0caa]
2021-02-02 15:57:35,857 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file
2021-02-02 15:57:35,857 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start
2021-02-02 15:57:35,857 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length
2021-02-02 15:57:35,857 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 1009
2021-02-02 15:57:37,109 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 268435452(1073741808)
2021-02-02 15:57:37,110 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 1024
2021-02-02 15:57:37,110 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 858993472
2021-02-02 15:57:37,110 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 1073741824
2021-02-02 15:57:37,110 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 268435452; length = 67108864
2021-02-02 15:57:37,114 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-02-02 15:57:37,119 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: conf classpath = [file:/data12/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/container_e03_1605975810748_1427_01_000559/, file:/data/sysdir/servers/hadoop-2.7.2-yarn/etc/hadoop/, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-nfs-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/mysql-connector-java-5.1.31-bin.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/spark-2.0.2-yarn-shuffle.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-common-2.7.2-tests.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/data-hadoop-patch.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-common-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common
/lib/jaxb-api-2.2.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-client-2.7.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/netty-3.6.2.Final.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jetty-6.1.26.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jetty-util-6.1.26.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/junit-4.11.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/avro-1.7.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jets3t-0.9.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-httpclient-3.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-math3-3.1.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/servl
et-api-2.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-logging-1.1.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/httpclient-4.2.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/protobuf-java-2.5.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/mockito-all-1.8.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hamcrest-core-1.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-recipes-2.7.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-collections-3.2.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-server-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.
2-yarn/share/hadoop/common/lib/guava-11.0.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hadoop-annotations-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/httpcore-4.2.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-io-2.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsp-api-2.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-json-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/zookeeper-3.4.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hadoop-auth-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-cli-1.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-core-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-lang-2.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/ha
doop/common/lib/slf4j-api-1.7.10.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/xmlenc-0.52.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jettison-1.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/api-util-1.0.0-M20.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/asm-3.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/snappy-java-1.0.4.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsch-0.1.42.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/stax-api-1.0-2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-xc-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/commo
n/lib/paranamer-2.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-configuration-1.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/log4j-1.2.17.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-compress-1.4.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/gson-2.2.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsr305-3.0.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-beanutils-1.7.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/java-xmlbuilder-0.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-net-3.1.jar, file:/data/sysdir/servers/hadoop-2.7.2
-yarn/share/hadoop/common/lib/commons-digester-1.8.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-framework-2.7.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/xz-1.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/activation-1.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-codec-1.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-2.7.2-tests.jar, file:/data/sysdir/servers/hadoop-2
.7.2-yarn/share/hadoop/hdfs/lib/ranger-plugin-classloader-1.0.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jetty-6.1.26.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/ranger-hdfs-plugin-shim-1.0.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/servlet-api-2.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jersey-server-1.9.jar, file:/data
/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/guava-11.0.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-io-2.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-cli-1.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jersey-core-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-lang-2.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xmlenc-0.52.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/asm-3.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoo
p/hdfs/lib/log4j-1.2.17.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jsr305-3.0.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-codec-1.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-client-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-common-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-
common-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-registry-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-api-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/netty-3.6.2.Final.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jetty-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jetty-util-6.1.26.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guice-3.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/servlet-api-2.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-logging-1.1.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guice-servlet-3.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-collections-3.2.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-server-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guava-11.0.2.jar, file:/data
/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/aopalliance-1.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-io-2.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-json-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-client-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/zookeeper-3.4.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-cli-1.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-core-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-lang-2.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jettison-1.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/asm-3.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/stax-api-1.0-2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yar
n/lib/jackson-xc-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/log4j-1.2.17.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/javax.inject-1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-compress-1.4.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jsr305-3.0.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-guice-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/xz-1.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/activation-1.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-codec-1.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-ya
rn/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/junit-4.11.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/avro-1.7.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/guice-3.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jersey-server-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/hadoop-annotation
s-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/aopalliance-1.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/commons-io-2.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jersey-core-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/asm-3.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/paranamer-2.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/log4j-1.2.17.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/javax.inject-1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduc
e/lib/jersey-guice-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/xz-1.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar, file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/job.jar, file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/classes, file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/lib/*, file:/data2/nm-local-dir/usercache/flink/filecache/299/UserAgentUtils-1.19.jar, file:/data9/nm-local-dir/usercache/flink/filecache/294/iceberg-hive-runtime-022dc46.dirty.jar, file:/data7/nm-local-dir/usercache/flink/filecache/292/hive-jdbc-1.2.1.jar, file:/data12/nm-local-dir/usercache/flink/filecache/297/dc-ua-util-2.0.0-SNAPSHOT.jar, file:/data11/nm-local-dir/usercache/flink/filecache/296/check-hive-db.jar, file:/data3/nm-local-dir/usercache/flink
/appcache/application_1605975810748_1427/filecache/10/job.jar/, file:/data10/nm-local-dir/usercache/flink/filecache/295/data-udf-0.0.1.jar, file:/data6/nm-local-dir/usercache/flink/filecache/291/sqljdbc4-4.0.jar]
2021-02-02 15:57:37,121 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper: thread classpath = [file:/data12/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/container_e03_1605975810748_1427_01_000559/, file:/data/sysdir/servers/hadoop-2.7.2-yarn/etc/hadoop/, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-nfs-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/mysql-connector-java-5.1.31-bin.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/spark-2.0.2-yarn-shuffle.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-common-2.7.2-tests.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/data-hadoop-patch.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-common-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/comm
on/lib/jaxb-api-2.2.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-client-2.7.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/netty-3.6.2.Final.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jetty-6.1.26.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jetty-util-6.1.26.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/junit-4.11.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/avro-1.7.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jets3t-0.9.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-httpclient-3.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-math3-3.1.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/ser
vlet-api-2.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-logging-1.1.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/httpclient-4.2.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/protobuf-java-2.5.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/mockito-all-1.8.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hamcrest-core-1.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-recipes-2.7.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-collections-3.2.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-server-1.9.jar, file:/data/sysdir/servers/hadoop-2.
7.2-yarn/share/hadoop/common/lib/guava-11.0.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hadoop-annotations-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/httpcore-4.2.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-io-2.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsp-api-2.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-json-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/zookeeper-3.4.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hadoop-auth-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-cli-1.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-core-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-lang-2.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/
hadoop/common/lib/slf4j-api-1.7.10.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/xmlenc-0.52.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jettison-1.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/api-util-1.0.0-M20.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/asm-3.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/snappy-java-1.0.4.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsch-0.1.42.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/stax-api-1.0-2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-xc-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/com
mon/lib/paranamer-2.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-configuration-1.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/log4j-1.2.17.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-compress-1.4.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/gson-2.2.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsr305-3.0.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-beanutils-1.7.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/java-xmlbuilder-0.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-net-3.1.jar, file:/data/sysdir/servers/hadoop-2.7
.2-yarn/share/hadoop/common/lib/commons-digester-1.8.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-framework-2.7.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/xz-1.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/activation-1.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-codec-1.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-2.7.2-tests.jar, file:/data/sysdir/servers/hadoop
-2.7.2-yarn/share/hadoop/hdfs/lib/ranger-plugin-classloader-1.0.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jetty-6.1.26.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/ranger-hdfs-plugin-shim-1.0.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/servlet-api-2.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jersey-server-1.9.jar, file:/da
ta/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/guava-11.0.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-io-2.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-cli-1.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jersey-core-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-lang-2.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xmlenc-0.52.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/asm-3.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/had
oop/hdfs/lib/log4j-1.2.17.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jsr305-3.0.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-codec-1.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-client-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-common-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-serve
r-common-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-registry-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-api-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/netty-3.6.2.Final.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jetty-6.1.26.jar
, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jetty-util-6.1.26.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guice-3.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/servlet-api-2.5.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-logging-1.1.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guice-servlet-3.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-collections-3.2.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-server-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guava-11.0.2.jar, file:/da
ta/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/aopalliance-1.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-io-2.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-json-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-client-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/zookeeper-3.4.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-cli-1.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-core-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-lang-2.6.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jettison-1.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/asm-3.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/stax-api-1.0-2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/y
arn/lib/jackson-xc-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/log4j-1.2.17.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/javax.inject-1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-compress-1.4.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jsr305-3.0.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-guice-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/xz-1.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/activation-1.1.ja
r, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-codec-1.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-
yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/junit-4.11.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/avro-1.7.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/guice-3.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jersey-server-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/hadoop-annotati
ons-2.7.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/aopalliance-1.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/commons-io-2.4.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jersey-core-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/asm-3.2.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/paranamer-2.3.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/log4j-1.2.17.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/javax.inject-1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapred
uce/lib/jersey-guice-1.9.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/xz-1.0.jar, file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar, file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/job.jar, file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/classes, file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/lib/*, file:/data2/nm-local-dir/usercache/flink/filecache/299/UserAgentUtils-1.19.jar, file:/data9/nm-local-dir/usercache/flink/filecache/294/iceberg-hive-runtime-022dc46.dirty.jar, file:/data7/nm-local-dir/usercache/flink/filecache/292/hive-jdbc-1.2.1.jar, file:/data12/nm-local-dir/usercache/flink/filecache/297/dc-ua-util-2.0.0-SNAPSHOT.jar, file:/data11/nm-local-dir/usercache/flink/filecache/296/check-hive-db.jar, file:/data3/nm-local-dir/usercache/fli
nk/appcache/application_1605975810748_1427/filecache/10/job.jar/, file:/data10/nm-local-dir/usercache/flink/filecache/295/data-udf-0.0.1.jar, file:/data6/nm-local-dir/usercache/flink/filecache/291/sqljdbc4-4.0.jar]
2021-02-02 15:57:37,121 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH = hdfs://ns6/tmp/hive/flink/25369d0b-110e-4eca-8553-f2f3953f0303/hive_2021-02-02_15-56-22_060_2459779186626852210-1/-mr-10006/33cfad1a-7384-4c27-b2d4-2f894ecb845a/map.xml
2021-02-02 15:57:37,123 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initializing operator MAP[0]
2021-02-02 15:57:37,128 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: serDeProperties : {columns.types=string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string, location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di, columns=version,load_id,pre_load_id,sign,visit_time,ip,province_id,city_id,county_id,appkey,session_id,user_id,ua,visits_id,first_visit_time,first_visit_load_id,first_visit_ref_domain,first_visit_subdomain_time,first_visit_subdomain_load_id,firs
t_visit_subdomain_refer,visits_ref,visits_keyword,visits_area_id,bt_visits_ref,last_visit_subdomain_refer,site_ref,scene_type,show_id,ref,refer_domain,key_word,area_id,current_url,current_domain,site_id,category_id,sub_category_id,object_id,series_id,spec_id,dealer_id,level_id,doc_title,page_custom_vars,platform,screen,color_depth,browser_charset,browser_language,browser_cookie_enable,extend,create_time,id,custom1,custom2,customer3,customer4,longitude,latitude,browse_type,ac_userid,ac_app_key,ac_app_platform,ac_app_ver,ac_app_deviceid,cur_visit_ref_domain,cookie_content,pre_visits_ref,lt_visits_ref,abtest,wx_openid,wx_source,log_src,dt,hour, metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00753-b2322cf0-ac32-4eb0-957e-95d2189e0272.metadata.json, previous_metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00752-6949a171-403e-4853-9
3a3-666df4305603.metadata.json, numFiles=59177, serialization.ddl=struct flink_fdm_iceberg_fdm_flw_pcm_log_clean_di { string version, string load_id, string pre_load_id, string sign, string visit_time, string ip, string province_id, string city_id, string county_id, string appkey, string session_id, string user_id, string ua, string visits_id, string first_visit_time, string first_visit_load_id, string first_visit_ref_domain, string first_visit_subdomain_time, string first_visit_subdomain_load_id, string first_visit_subdomain_refer, string visits_ref, string visits_keyword, string visits_area_id, string bt_visits_ref, string last_visit_subdomain_refer, string site_ref, string scene_type, string show_id, string ref, string refer_domain, string key_word, string area_id, string current_url, string current_domain, string site_id, string category_id, string sub_category_id, string object_id, string series_id, string spec_id, string dealer_id, string level_id, string doc_title, string pag
e_custom_vars, string platform, string screen, string color_depth, string browser_charset, string browser_language, string browser_cookie_enable, string extend, string create_time, string id, string custom1, string custom2, string customer3, string customer4, string longitude, string latitude, string browse_type, string ac_userid, string ac_app_key, string ac_app_platform, string ac_app_ver, string ac_app_deviceid, string cur_visit_ref_domain, string cookie_content, string pre_visits_ref, string lt_visits_ref, string abtest, string wx_openid, string wx_source, string log_src, string dt, string hour}, transient_lastDdlTime=1611904166, iceberg.mr.table.schema={"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id"
:6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"first_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,
"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required":false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,
"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform","required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"cre
ate_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"i
d":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]}, columns.comments=, totalSize=302458258710, bucket_count=0, file.outputformat=org.apache.iceberg.mr.hive.HiveIcebergOutputFormat, serialization.lib=org.apache.iceberg.mr.hive.HiveIcebergSerDe, storage_handler=org.apache.iceberg.mr.hive.HiveIcebergStorageHandler, table_type=ICEBERG, file.inputformat=org.apache.iceberg.mr.hive.HiveIcebergInputFormat, EXTERNAL=TRUE, name=flink_fdm_iceberg.flink_fdm_iceberg_fdm_flw_pcm_log_clean_di}
2021-02-02 15:57:37,906 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using schema from serDeProperties {"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id":6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"f
irst_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required"
:false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform"
,"required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"create_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac
_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"id":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]}
2021-02-02 15:57:37,907 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: selectedColumns 1, [wx_source]
2021-02-02 15:57:37,926 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using projectedSchema {"type":"struct","fields":[{"id":72,"name":"wx_source","required":false,"type":"string"}]}
2021-02-02 15:57:37,969 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: serDeProperties : {columns.types=string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string, location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di, columns=version,load_id,pre_load_id,sign,visit_time,ip,province_id,city_id,county_id,appkey,session_id,user_id,ua,visits_id,first_visit_time,first_visit_load_id,first_visit_ref_domain,first_visit_subdomain_time,first_visit_subdomain_load_id,firs
t_visit_subdomain_refer,visits_ref,visits_keyword,visits_area_id,bt_visits_ref,last_visit_subdomain_refer,site_ref,scene_type,show_id,ref,refer_domain,key_word,area_id,current_url,current_domain,site_id,category_id,sub_category_id,object_id,series_id,spec_id,dealer_id,level_id,doc_title,page_custom_vars,platform,screen,color_depth,browser_charset,browser_language,browser_cookie_enable,extend,create_time,id,custom1,custom2,customer3,customer4,longitude,latitude,browse_type,ac_userid,ac_app_key,ac_app_platform,ac_app_ver,ac_app_deviceid,cur_visit_ref_domain,cookie_content,pre_visits_ref,lt_visits_ref,abtest,wx_openid,wx_source,log_src,dt,hour, metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00753-b2322cf0-ac32-4eb0-957e-95d2189e0272.metadata.json, previous_metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00752-6949a171-403e-4853-9
3a3-666df4305603.metadata.json, numFiles=59177, serialization.ddl=struct flink_fdm_iceberg_fdm_flw_pcm_log_clean_di { string version, string load_id, string pre_load_id, string sign, string visit_time, string ip, string province_id, string city_id, string county_id, string appkey, string session_id, string user_id, string ua, string visits_id, string first_visit_time, string first_visit_load_id, string first_visit_ref_domain, string first_visit_subdomain_time, string first_visit_subdomain_load_id, string first_visit_subdomain_refer, string visits_ref, string visits_keyword, string visits_area_id, string bt_visits_ref, string last_visit_subdomain_refer, string site_ref, string scene_type, string show_id, string ref, string refer_domain, string key_word, string area_id, string current_url, string current_domain, string site_id, string category_id, string sub_category_id, string object_id, string series_id, string spec_id, string dealer_id, string level_id, string doc_title, string pag
e_custom_vars, string platform, string screen, string color_depth, string browser_charset, string browser_language, string browser_cookie_enable, string extend, string create_time, string id, string custom1, string custom2, string customer3, string customer4, string longitude, string latitude, string browse_type, string ac_userid, string ac_app_key, string ac_app_platform, string ac_app_ver, string ac_app_deviceid, string cur_visit_ref_domain, string cookie_content, string pre_visits_ref, string lt_visits_ref, string abtest, string wx_openid, string wx_source, string log_src, string dt, string hour}, transient_lastDdlTime=1611904166, iceberg.mr.table.schema={"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id"
:6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"first_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,
"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required":false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,
"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform","required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"cre
ate_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"i
d":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]}, columns.comments=, totalSize=302458258710, bucket_count=0, file.outputformat=org.apache.iceberg.mr.hive.HiveIcebergOutputFormat, serialization.lib=org.apache.iceberg.mr.hive.HiveIcebergSerDe, storage_handler=org.apache.iceberg.mr.hive.HiveIcebergStorageHandler, table_type=ICEBERG, file.inputformat=org.apache.iceberg.mr.hive.HiveIcebergInputFormat, EXTERNAL=TRUE, name=flink_fdm_iceberg.flink_fdm_iceberg_fdm_flw_pcm_log_clean_di}
2021-02-02 15:57:37,973 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using schema from serDeProperties {"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id":6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"f
irst_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required"
:false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform"
,"required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"create_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac
_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"id":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]}
2021-02-02 15:57:37,974 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: selectedColumns 1, [wx_source]
2021-02-02 15:57:37,974 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using projectedSchema {"type":"struct","fields":[{"id":72,"name":"wx_source","required":false,"type":"string"}]}
2021-02-02 15:57:37,998 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: serDeProperties : {columns.types=string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string, location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di, columns=version,load_id,pre_load_id,sign,visit_time,ip,province_id,city_id,county_id,appkey,session_id,user_id,ua,visits_id,first_visit_time,first_visit_load_id,first_visit_ref_domain,first_visit_subdomain_time,first_visit_subdomain_load_id,firs
t_visit_subdomain_refer,visits_ref,visits_keyword,visits_area_id,bt_visits_ref,last_visit_subdomain_refer,site_ref,scene_type,show_id,ref,refer_domain,key_word,area_id,current_url,current_domain,site_id,category_id,sub_category_id,object_id,series_id,spec_id,dealer_id,level_id,doc_title,page_custom_vars,platform,screen,color_depth,browser_charset,browser_language,browser_cookie_enable,extend,create_time,id,custom1,custom2,customer3,customer4,longitude,latitude,browse_type,ac_userid,ac_app_key,ac_app_platform,ac_app_ver,ac_app_deviceid,cur_visit_ref_domain,cookie_content,pre_visits_ref,lt_visits_ref,abtest,wx_openid,wx_source,log_src,dt,hour, metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00753-b2322cf0-ac32-4eb0-957e-95d2189e0272.metadata.json, previous_metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00752-6949a171-403e-4853-9
3a3-666df4305603.metadata.json, numFiles=59177, serialization.ddl=struct flink_fdm_iceberg_fdm_flw_pcm_log_clean_di { string version, string load_id, string pre_load_id, string sign, string visit_time, string ip, string province_id, string city_id, string county_id, string appkey, string session_id, string user_id, string ua, string visits_id, string first_visit_time, string first_visit_load_id, string first_visit_ref_domain, string first_visit_subdomain_time, string first_visit_subdomain_load_id, string first_visit_subdomain_refer, string visits_ref, string visits_keyword, string visits_area_id, string bt_visits_ref, string last_visit_subdomain_refer, string site_ref, string scene_type, string show_id, string ref, string refer_domain, string key_word, string area_id, string current_url, string current_domain, string site_id, string category_id, string sub_category_id, string object_id, string series_id, string spec_id, string dealer_id, string level_id, string doc_title, string pag
e_custom_vars, string platform, string screen, string color_depth, string browser_charset, string browser_language, string browser_cookie_enable, string extend, string create_time, string id, string custom1, string custom2, string customer3, string customer4, string longitude, string latitude, string browse_type, string ac_userid, string ac_app_key, string ac_app_platform, string ac_app_ver, string ac_app_deviceid, string cur_visit_ref_domain, string cookie_content, string pre_visits_ref, string lt_visits_ref, string abtest, string wx_openid, string wx_source, string log_src, string dt, string hour}, transient_lastDdlTime=1611904166, iceberg.mr.table.schema={"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id"
:6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"first_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,
"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required":false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,
"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform","required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"cre
ate_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"i
d":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]}, columns.comments=, totalSize=302458258710, bucket_count=0, file.outputformat=org.apache.iceberg.mr.hive.HiveIcebergOutputFormat, serialization.lib=org.apache.iceberg.mr.hive.HiveIcebergSerDe, storage_handler=org.apache.iceberg.mr.hive.HiveIcebergStorageHandler, table_type=ICEBERG, file.inputformat=org.apache.iceberg.mr.hive.HiveIcebergInputFormat, EXTERNAL=TRUE, name=flink_fdm_iceberg.flink_fdm_iceberg_fdm_flw_pcm_log_clean_di}
2021-02-02 15:57:37,999 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using schema from serDeProperties {"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id":6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"f
irst_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required"
:false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform"
,"required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"create_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac
_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"id":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]}
2021-02-02 15:57:37,999 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: selectedColumns 1, [wx_source]
2021-02-02 15:57:38,000 INFO [main] org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using projectedSchema {"type":"struct","fields":[{"id":72,"name":"wx_source","required":false,"type":"string"}]}
2021-02-02 15:57:38,005 INFO [main] org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
<MAP>Id =0
<Children>null
<TS>Id =3
<Children>
<FIL>Id =18
<Children>
<SEL>Id =5
<Children>
<RS>Id =7
<Children>
<\Children>
<Parent>Id = 5 null<\Parent>
<\RS>
<\Children>
<Parent>Id = 18 null<\Parent>
<\SEL>
<\Children>
<Parent>Id = 3 null<\Parent>
<\FIL>
<\Children>
<Parent>Id = 0 null<\Parent>
<\TS>
<\Children>
<Parent><\Parent>
<\MAP>
2021-02-02 15:57:38,007 INFO [main] org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[3]
2021-02-02 15:57:38,008 INFO [main] org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing operator FIL[18]
2021-02-02 15:57:43,031 INFO [main] org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing operator SEL[5]
2021-02-02 15:57:43,031 INFO [main] org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT struct<version:string,load_id:string,pre_load_id:string,sign:string,visit_time:string,ip:string,province_id:int,city_id:int,county_id:string,appkey:string,session_id:string,user_id:int,ua:string,visits_id:string,first_visit_time:string,first_visit_load_id:string,first_visit_ref_domain:string,first_visit_subdomain_time:string,first_visit_subdomain_load_id:string,first_visit_subdomain_refer:string,visits_ref:string,visits_keyword:string,visits_area_id:string,bt_visits_ref:string,last_visit_subdomain_refer:string,site_ref:string,scene_type:string,show_id:string,ref:string,refer_domain:string,key_word:string,area_id:string,current_url:string,current_domain:string,site_id:int,category_id:int,sub_category_id:int,object_id:int,series_id:int,spec_id:int,dealer_id:int,level_id:int,doc_title:string,page_custom_vars:string,platform:int,screen:string,color_depth:string,browser_charset:string,browser_lang
uage:string,browser_cookie_enable:string,extend:string,create_time:string,id:string,custom1:int,custom2:int,customer3:string,customer4:string,longitude:float,latitude:float,browse_type:string,ac_userid:string,ac_app_key:string,ac_app_platform:string,ac_app_ver:string,ac_app_deviceid:string,cur_visit_ref_domain:string,cookie_content:string,pre_visits_ref:string,lt_visits_ref:string,abtest:string,wx_openid:string,wx_source:string,log_src:string,dt:string,hour:string>
2021-02-02 15:57:43,033 INFO [main] org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing operator RS[7]
2021-02-02 15:57:43,036 INFO [main] org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Using tag = 1
2021-02-02 15:57:43,050 INFO [main] org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[0]
2021-02-02 15:57:43,050 INFO [main] org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing operator FIL[17]
2021-02-02 15:57:43,051 INFO [main] org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing operator SEL[2]
2021-02-02 15:57:43,051 INFO [main] org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT struct<wx_source:string>
2021-02-02 15:57:43,052 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:136)
... 22 more
Caused by: java.lang.RuntimeException: cannot find field log_src from [org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObjectInspector$IcebergRecordStructField@8838736]
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:442)
at org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObjectInspector.getStructFieldRef(IcebergRecordObjectInspector.java:78)
at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:139)
at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:980)
at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:1006)
at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:77)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:498)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:115)
... 22 more
2021-02-02 15:57:43,053 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dixingxing0 commented on issue #2198: Hive: got error while joining iceberg table and hive table
Posted by GitBox <gi...@apache.org>.
dixingxing0 commented on issue #2198:
URL: https://github.com/apache/iceberg/issues/2198#issuecomment-810067302
Sorry for the late reply, i found that this exception only occurs in the initializeMapOperator phase, org.apache.iceberg.mr.hive.HiveIcebergSerDe will not be used to deserialize data from hive table, so i just simply caught this exception and logged it, then everything works fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2198: Hive: got error while joining iceberg table and hive table
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2198:
URL: https://github.com/apache/iceberg/issues/2198#issuecomment-771636006
@dixingxing0: I have tried this in the unit tests where the customers table is an iceberg table and have 3 rows:
```
CREATE TABLE c2 (customer_id bigint, first_name string, last_name string);
select count(1),count(a.customer_id),count(b.customer_id)
from (select customer_id, concat_ws(first_name, last_name) as str from default.customers where customer_id>0) a
full outer join (select customer_id, concat_ws(first_name, last_name) as str from default.c2 where customer_id>0) b
on a.str=b.str
where a.str is null or b.str is null;
```
The queries were successfully executed on my env (Hive 2.3.7/3.1.2, Iceberg master branch)
To repro the case there are several things that would be really helpful:
- `SHOW CREATE TABLE` output for both source tables
- Could you please check if the `empty` file actually in the `hive_table1` directory?
- Could you provide some test data for the iceberg_table?
- Could you please try to simplify the query?
Thanks,
Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dixingxing0 closed issue #2198: Hive: got error while joining iceberg table and hive table
Posted by GitBox <gi...@apache.org>.
dixingxing0 closed issue #2198:
URL: https://github.com/apache/iceberg/issues/2198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #2198: Hive: got error while joining iceberg table and hive table
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #2198:
URL: https://github.com/apache/iceberg/issues/2198#issuecomment-771636006
@dixingxing0: I have tried this in the unit tests where the customers table is an iceberg table and have 3 rows:
```
CREATE TABLE c2 (customer_id bigint, first_name string, last_name string);
select count(1),count(a.customer_id),count(b.customer_id)
from (select customer_id, concat_ws(first_name, last_name) as str from default.customers where customer_id>0) a
full outer join (select customer_id, concat_ws(first_name, last_name) as str from default.c2 where customer_id>0) b
on a.str=b.str
where a.str is null or b.str is null;
```
The queries were successfully executed on my env (Hive 2.3.7/3.1.2, Iceberg master branch)
To repro the case there are several things that would be really helpful:
- `SHOW CREATE TABLE` output for both source tables
- Could you please check if the `empty` file actually in the `hive_table1` directory?
- Could you provide some test data for the iceberg_table?
- Could you please try to simplify the query?
Thanks,
Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org