You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@linkis.apache.org by GitBox <gi...@apache.org> on 2022/09/09 09:44:51 UTC
[GitHub] [incubator-linkis] 2018yinjian commented on issue #3303: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field
2018yinjian commented on issue #3303:
URL: https://github.com/apache/incubator-linkis/issues/3303#issuecomment-1241750642
spark-submit --master yarn test.py这种方式提交也是可以读取kudu表的
cat test.py
import os,sys,linecache,commands
import datetime
from pyspark.sql import SQLContext,Row,SparkSession
from pyspark.sql.types import *
from pyspark.sql import HiveContext
reload(sys)
sys.setdefaultencoding("utf-8")
def readKudu(table_name):
df = spark.read.format("org.apache.kudu.spark.kudu") \
.option("kudu.table", table_name) \
.option("kudu.master", "192.168.217.248") \
.load()
df.show()
df.printSchema()
def init_spark_context():
spark = SparkSession.builder.master("yarn-client").appName("testApp").enableHiveSupport().getOrCreate()
return spark
if __name__ == "__main__":
spark = init_spark_context()
readKudu("impala::dim.dim_src_tab_info")
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/apps/ecm/service/spark/2.4.7-hadoop3.2-1.1.1/package/spark-2.4.7-hadoop3.2-1.1.1/jars/ess-shuffle-manager-1.0.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/apps/ecm/service/spark/2.4.7-hadoop3.2-1.1.1/package/spark-2.4.7-hadoop3.2-1.1.1/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
22/09/09 17:36:59 INFO [Thread-5] SparkContext: Running Spark version 2.4.7
22/09/09 17:36:59 WARN [Thread-5] SparkConf: spark.master yarn-client is deprecated in Spark 2.0+, please instead use "yarn" with specified deploy mode.
22/09/09 17:36:59 INFO [Thread-5] SparkContext: Submitted application: testApp
22/09/09 17:36:59 INFO [Thread-5] SecurityManager: Changing view acls to: yinjian,*
22/09/09 17:36:59 INFO [Thread-5] SecurityManager: Changing modify acls to: yinjian
22/09/09 17:36:59 INFO [Thread-5] SecurityManager: Changing view acls groups to:
22/09/09 17:36:59 INFO [Thread-5] SecurityManager: Changing modify acls groups to:
22/09/09 17:36:59 INFO [Thread-5] SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yinjian, *); groups with view permissions: Set(); users with modify permissions: Set(yinjian); groups with modify permissions: Set()
22/09/09 17:36:59 INFO [Thread-5] Utils: Successfully started service 'sparkDriver' on port 13247.
22/09/09 17:36:59 INFO [Thread-5] SparkEnv: Registering MapOutputTracker
22/09/09 17:36:59 INFO [Thread-5] SparkEnv: Registering BlockManagerMaster
22/09/09 17:36:59 INFO [Thread-5] BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/09/09 17:36:59 INFO [Thread-5] BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/09/09 17:36:59 INFO [Thread-5] DiskBlockManager: Created local directory at /tmp/blockmgr-6b896b1f-0b53-4651-a807-367e14e9fd97
22/09/09 17:36:59 INFO [Thread-5] MemoryStore: MemoryStore started with capacity 4.1 GB
22/09/09 17:36:59 INFO [Thread-5] SparkEnv: Registering OutputCommitCoordinator
22/09/09 17:37:00 WARN [Thread-5] Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
22/09/09 17:37:00 INFO [Thread-5] Utils: Successfully started service 'SparkUI' on port 4041.
22/09/09 17:37:00 INFO [Thread-5] SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.217.247:4041
22/09/09 17:37:00 INFO [Thread-5] Client: Requesting a new application from cluster with 5 NodeManagers
22/09/09 17:37:01 INFO [Thread-5] Configuration: resource-types.xml not found
22/09/09 17:37:01 INFO [Thread-5] ResourceUtils: Unable to find 'resource-types.xml'.
22/09/09 17:37:01 INFO [Thread-5] Client: Verifying our application has not requested more than the maximum memory capability of the cluster (30720 MB per container)
22/09/09 17:37:01 INFO [Thread-5] Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
22/09/09 17:37:01 INFO [Thread-5] Client: Setting up container launch context for our AM
22/09/09 17:37:01 INFO [Thread-5] Client: Setting up the launch environment for our AM container
22/09/09 17:37:01 INFO [Thread-5] Client: Preparing resources for our AM container
22/09/09 17:37:01 WARN [Thread-5] Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
22/09/09 17:37:05 INFO [Thread-5] Client: Uploading resource file:/tmp/spark-296f02e9-cd3a-4894-a78d-130953efd23b/__spark_libs__7120621387282183291.zip -> hdfs://emr-cluster/user/yinjian/.sparkStaging/application_1640111027208_230172/__spark_libs__7120621387282183291.zip
22/09/09 17:37:05 INFO [Thread-25] SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
22/09/09 17:37:05 INFO [DataStreamer for file /user/yinjian/.sparkStaging/application_1640111027208_230172/__spark_libs__7120621387282183291.zip] SaslDataTransferClient: SASL encryption trust check: localHostTrusted= false, remoteHostTrusted = false
22/09/09 17:37:05 INFO [DataStreamer for file /user/yinjian/.sparkStaging/application_1640111027208_230172/__spark_libs__7120621387282183291.zip] SaslDataTransferClient: SASL encryption trust check: localHostTrusted= false, remoteHostTrusted = false
22/09/09 17:37:06 INFO [Thread-5] Client: Uploading resource file:/usr/lib/spark-current/python/lib/pyspark.zip -> hdfs://emr-cluster/user/yinjian/.sparkStaging/application_1640111027208_230172/pyspark.zip
22/09/09 17:37:06 INFO [Thread-30] SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
22/09/09 17:37:06 INFO [Thread-5] Client: Uploading resource file:/usr/lib/spark-current/python/lib/py4j-0.10.7-src.zip -> hdfs://emr-cluster/user/yinjian/.sparkStaging/application_1640111027208_230172/py4j-0.10.7-src.zip
22/09/09 17:37:06 INFO [Thread-32] SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
22/09/09 17:37:07 INFO [Thread-5] Client: Uploading resource file:/tmp/spark-296f02e9-cd3a-4894-a78d-130953efd23b/__spark_conf__4559551404591797490.zip -> hdfs://emr-cluster/user/yinjian/.sparkStaging/application_1640111027208_230172/__spark_conf__.zip
22/09/09 17:37:07 INFO [Thread-34] SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
22/09/09 17:37:07 INFO [Thread-5] Client: EMR SmartData using Java binary at JAVA_HOME.
22/09/09 17:37:07 INFO [Thread-5] SecurityManager: Changing view acls to: yinjian,*
22/09/09 17:37:07 INFO [Thread-5] SecurityManager: Changing modify acls to: yinjian
22/09/09 17:37:07 INFO [Thread-5] SecurityManager: Changing view acls groups to:
22/09/09 17:37:07 INFO [Thread-5] SecurityManager: Changing modify acls groups to:
22/09/09 17:37:07 INFO [Thread-5] SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yinjian, *); groups with view permissions: Set(); users with modify permissions: Set(yinjian); groups with modify permissions: Set()
22/09/09 17:37:07 INFO [Thread-5] EsServiceCredentialProvider: Loaded EsServiceCredentialProvider
22/09/09 17:37:07 INFO [Thread-5] EsServiceCredentialProvider: Hadoop Security Enabled = [false]
22/09/09 17:37:07 INFO [Thread-5] EsServiceCredentialProvider: ES Auth Method = [SIMPLE]
22/09/09 17:37:07 INFO [Thread-5] EsServiceCredentialProvider: Are creds required = [false]
22/09/09 17:37:07 INFO [Thread-5] Client: Submitting application application_1640111027208_230172 to ResourceManager
22/09/09 17:37:08 INFO [Thread-5] YarnClientImpl: Submitted application application_1640111027208_230172
22/09/09 17:37:08 INFO [Thread-5] SchedulerExtensionServices: Starting Yarn extension services with app application_1640111027208_230172 and attemptId None
22/09/09 17:37:09 INFO [Thread-5] Client: Application report for application_1640111027208_230172 (state: ACCEPTED)
22/09/09 17:37:09 INFO [Thread-5] Client:
client token: N/A
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1662716227968
final status: UNDEFINED
tracking URL: http://emr-header-1.cluster-254539:20888/proxy/application_1640111027208_230172/
user: yinjian
22/09/09 17:37:10 INFO [Thread-5] Client: Application report for application_1640111027208_230172 (state: ACCEPTED)
22/09/09 17:37:11 INFO [Thread-5] Client: Application report for application_1640111027208_230172 (state: ACCEPTED)
22/09/09 17:37:12 INFO [Thread-5] Client: Application report for application_1640111027208_230172 (state: ACCEPTED)
22/09/09 17:37:13 INFO [Thread-5] Client: Application report for application_1640111027208_230172 (state: ACCEPTED)
22/09/09 17:37:14 INFO [Thread-5] Client: Application report for application_1640111027208_230172 (state: ACCEPTED)
22/09/09 17:37:15 INFO [dispatcher-event-loop-10] YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> emr-header-1.cluster-254539, PROXY_URI_BASES -> http://emr-header-1.cluster-254539:20888/proxy/application_1640111027208_230172, RM_HA_URLS -> emr-header-1.cluster-254539:8088,emr-header-2.cluster-254539:8088), /proxy/application_1640111027208_230172
22/09/09 17:37:15 INFO [Thread-5] Client: Application report for application_1640111027208_230172 (state: RUNNING)
22/09/09 17:37:15 INFO [Thread-5] Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.217.249
ApplicationMaster RPC port: -1
queue: default
start time: 1662716227968
final status: UNDEFINED
tracking URL: http://emr-header-1.cluster-254539:20888/proxy/application_1640111027208_230172/
user: yinjian
22/09/09 17:37:15 INFO [Thread-5] YarnClientSchedulerBackend: Application application_1640111027208_230172 has started running.
22/09/09 17:37:15 INFO [Thread-5] Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 19463.
22/09/09 17:37:15 INFO [Thread-5] NettyBlockTransferService: Server created on 192.168.217.247:19463
22/09/09 17:37:15 INFO [Thread-5] BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/09/09 17:37:15 INFO [Thread-5] BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.217.247, 19463, None)
22/09/09 17:37:15 INFO [dispatcher-event-loop-13] BlockManagerMasterEndpoint: Registering block manager 192.168.217.247:19463 with 4.1 GB RAM, BlockManagerId(driver, 192.168.217.247, 19463, None)
22/09/09 17:37:15 INFO [Thread-5] BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.217.247, 19463, None)
22/09/09 17:37:15 INFO [Thread-5] BlockManager: external shuffle service port = 7337
22/09/09 17:37:15 INFO [Thread-5] BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.217.247, 19463, None)
22/09/09 17:37:15 INFO [dispatcher-event-loop-14] YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
22/09/09 17:37:15 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /metrics/json.
22/09/09 17:37:15 INFO [Thread-5] EventLoggingListener: Logging events to hdfs://emr-cluster/spark-history/application_1640111027208_230172
22/09/09 17:37:15 INFO [Thread-37] SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
22/09/09 17:37:22 INFO [dispatcher-event-loop-5] YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.218.173:57104) with ID 2
22/09/09 17:37:22 INFO [dispatcher-event-loop-0] BlockManagerMasterEndpoint: Registering block manager emr-worker-5.cluster-254539:44863 with 3.4 GB RAM, BlockManagerId(2, emr-worker-5.cluster-254539, 44863, None)
22/09/09 17:37:24 INFO [dispatcher-event-loop-5] YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.217.250:15012) with ID 1
22/09/09 17:37:24 INFO [Thread-5] YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
22/09/09 17:37:24 INFO [dispatcher-event-loop-13] BlockManagerMasterEndpoint: Registering block manager emr-worker-2.cluster-254539:22195 with 3.4 GB RAM, BlockManagerId(1, emr-worker-2.cluster-254539, 22195, None)
22/09/09 17:37:24 INFO [Thread-5] SharedState: loading hive config file: file:/etc/ecm/spark-conf-2.4.7-hadoop3.2-1.1.1/hive-site.xml
22/09/09 17:37:24 INFO [Thread-5] SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
22/09/09 17:37:24 INFO [Thread-5] SharedState: Warehouse path is '/user/hive/warehouse'.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/json.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /SQL/execution/json.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql/json.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql/statistics.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql/statistics/json.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /static/sql.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql/query/kill.
22/09/09 17:37:24 INFO [Thread-5] JettyUtils: Adding filter org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter to /streamingsql/query/restart.
22/09/09 17:37:25 INFO [Thread-5] StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
22/09/09 17:37:25 INFO [Thread-5] Version: Elasticsearch Hadoop v7.10.2 [f53f4b7b2b]
22/09/09 17:37:27 INFO [Thread-5] CodeGenerator: Code generated in 206.944333 ms
22/09/09 17:37:27 INFO [Thread-5] ResultQueryStage: add exchangecoordinator though
22/09/09 17:37:27 INFO [Thread-5] CodeGenerator: Code generated in 21.478254 ms
22/09/09 17:37:27 INFO [Thread-5] SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0
22/09/09 17:37:27 INFO [dag-scheduler-event-loop] DAGScheduler: Got job 0 (showString at NativeMethodAccessorImpl.java:0) with 1 output partitions
22/09/09 17:37:27 INFO [dag-scheduler-event-loop] DAGScheduler: Final stage: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0)
22/09/09 17:37:27 INFO [dag-scheduler-event-loop] DAGScheduler: Parents of final stage: List()
22/09/09 17:37:27 INFO [dag-scheduler-event-loop] DAGScheduler: Missing parents: List()
22/09/09 17:37:27 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[4] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents
22/09/09 17:37:27 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_0 stored as values in memory (estimated size 28.2 KB, free 4.1 GB)
22/09/09 17:37:27 INFO [dag-scheduler-event-loop] MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 6.3 KB, free 4.1 GB)
22/09/09 17:37:27 INFO [dispatcher-event-loop-4] BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.217.247:19463 (size: 6.3 KB, free: 4.1 GB)
22/09/09 17:37:27 INFO [dag-scheduler-event-loop] SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1184
22/09/09 17:37:27 INFO [dag-scheduler-event-loop] DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[4] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitionsVector(0))
22/09/09 17:37:27 INFO [dag-scheduler-event-loop] YarnScheduler: Adding task set 0.0 with 1 tasks
22/09/09 17:37:27 INFO [dispatcher-event-loop-9] TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, emr-worker-5.cluster-254539, executor 2, partition 0, RACK_LOCAL, 8601 bytes)
22/09/09 17:37:27 INFO [dispatcher-event-loop-1] BlockManagerInfo: Added broadcast_0_piece0 in memory on emr-worker-5.cluster-254539:44863 (size: 6.3 KB, free: 3.4 GB)
22/09/09 17:37:30 INFO [task-result-getter-0] TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 2615 ms on emr-worker-5.cluster-254539 (executor 2) (1/1)
22/09/09 17:37:30 INFO [task-result-getter-0] YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool
22/09/09 17:37:30 INFO [dag-scheduler-event-loop] DAGScheduler: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0) finished in 2.813 s
22/09/09 17:37:30 INFO [Thread-5] DAGScheduler: Job 0 finished: showString at NativeMethodAccessorImpl.java:0, took 2.855944 s
22/09/09 17:37:30 INFO [Thread-5] SparkSQLQueryListener: head is called
22/09/09 17:37:30 INFO [Thread-5] SparkSQLQueryListener: Spark user yinjian executed on 1662716250192 with spark sql successfully.
+--------------------+--------+--------+--------------------+--------+------------+--------------+--------------------+--------------------+--------+--------+
| tar_tab|tar_part|src_inst| src_tab|src_part| src_pk|src_filter_col| cols| create_sql|batch_id|dim_flag|
+--------------------+--------+--------+--------------------+--------+------------+--------------+--------------------+--------------------+--------+--------+
|dim.dim_abnormal_...| | mysql57|zmn_ums.abnor_cat...| | categ_id| update_time|`name`,`first_let...|create table if n...| 9| 1|
| dim.dim_account| | mysql56| zmn_account.account| | account_id| update_time|`plat`,`account_c...|create table if n...| 9| 1|
| dim.dim_act_subject| | mysql57| zmn_act.act_subject| | subject_id| update_time|`name`,`show_name...|create table if n...| 9| 1|
|dim.dim_aws_base_...| | mysql56|zmn_aws.base_comm...| |community_id| update_time|`community_name`,...|create table if n...| 9| 1|
|dim.dim_aws_commu...| | mysql56|zmn_aws.aws_commu...| | id| update_time|`tripartite_commu...|create table if n...| 9| 1|
|dim.dim_base_channel| | mysql56|base_channel.channel| | channel_id| update_time|`name`,`name_piny...|create table if n...| 5| 2|
|dim.dim_base_chan...| | mysql56|base_channel.chan...| | channel_id| update_time|`name`,`salesman_...|create table if n...| 5| 2|
|dim.dim_base_chan...| | mysql56|base_channel.coop...| | relate_id| |`cooperate_id`,`t...|create table if n...| 5| 2|
|dim.dim_base_chan...| | mysql56|base_channel.chan...| | mark_id| update_time|`name`,`sort`,`st...|create table if n...| 5| 2|
|dim.dim_base_chan...| | mysql56|base_channel.chan...| | relate_id| update_time|`mark_id`,`channe...|create table if n...| 5| 2|
|dim.dim_base_chan...| | mysql56|base_channel.chan...| | channel_id| update_time|`province_id`,`ci...|create table if n...| 5| 2|
|dim.dim_base_comm...| | mysql56|base_common_data....| | area_id| update_time|`name`,`code`,`ar...|create table if n...| 5| 2|
|dim.dim_base_comm...| | mysql56|base_common_data....| | channel_id| update_time|`name`,`name_piny...|create table if n...| 5| 2|
|dim.dim_base_comm...| | mysql56|base_common_data....| | company_id| update_time|`company_code`,`n...|create table if n...| 5| 2|
|dim.dim_base_comm...| | mysql56|base_common_data....| | company_id| update_time|`full_name`,`lega...|create table if n...| 5| 2|
|dim.dim_base_comm...| | mysql56|base_common_data....| | company_id| update_time|`full_name`,`prov...|create table if n...| 5| 2|
|dim.dim_base_comm...| | mysql56|base_common_data....| | company_id| update_time|`full_name`,`bank...|create table if n...| 5| 2|
|dim.dim_base_comm...| | mysql56|base_common_data....| | gate_id| update_time|`gate_name`,`gate...|create table if n...| 5| 2|
|dim.dim_base_comm...| | mysql56|base_common_data....| | account_id| update_time|`name`,`account_t...|create table if n...| 5| 2|
|dim.dim_base_comm...| | mysql56|base_common_data....| | brand_id| update_time|`name`,`eng_name`...|create table if n...| 5| 2|
+--------------------+--------+--------+--------------------+--------+------------+--------------+--------------------+--------------------+--------+--------+
only showing top 20 rows
root
|-- tar_tab: string (nullable = false)
|-- tar_part: string (nullable = true)
|-- src_inst: string (nullable = true)
|-- src_tab: string (nullable = true)
|-- src_part: string (nullable = true)
|-- src_pk: string (nullable = true)
|-- src_filter_col: string (nullable = true)
|-- cols: string (nullable = true)
|-- create_sql: string (nullable = true)
|-- batch_id: integer (nullable = true)
|-- dim_flag: short (nullable = true)
22/09/09 17:37:30 INFO [shutdown-hook-0] SparkContext: Invoking stop() from shutdown hook
22/09/09 17:37:30 INFO [shutdown-hook-0] SparkUI: Stopped Spark web UI at http://192.168.217.247:4041
22/09/09 17:37:30 INFO [YARN application state monitor] YarnClientSchedulerBackend: Interrupting monitor thread
22/09/09 17:37:30 INFO [shutdown-hook-0] YarnClientSchedulerBackend: Shutting down all executors
22/09/09 17:37:30 INFO [dispatcher-event-loop-1] YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
22/09/09 17:37:30 INFO [shutdown-hook-0] SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
22/09/09 17:37:30 INFO [shutdown-hook-0] YarnClientSchedulerBackend: Stopped
22/09/09 17:37:30 INFO [dispatcher-event-loop-4] MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/09/09 17:37:30 INFO [shutdown-hook-0] MemoryStore: MemoryStore cleared
22/09/09 17:37:30 INFO [shutdown-hook-0] BlockManager: BlockManager stopped
22/09/09 17:37:30 INFO [shutdown-hook-0] BlockManagerMaster: BlockManagerMaster stopped
22/09/09 17:37:30 INFO [dispatcher-event-loop-7] OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/09/09 17:37:30 INFO [shutdown-hook-0] SparkContext: Successfully stopped SparkContext
22/09/09 17:37:30 INFO [shutdown-hook-0] ShutdownHookManager: Shutdown hook called
22/09/09 17:37:30 INFO [shutdown-hook-0] ShutdownHookManager: Deleting directory /tmp/spark-296f02e9-cd3a-4894-a78d-130953efd23b
22/09/09 17:37:30 INFO [shutdown-hook-0] ShutdownHookManager: Deleting directory /tmp/spark-296f02e9-cd3a-4894-a78d-130953efd23b/pyspark-1e4cbe5a-fc49-4f91-aa0c-b0371188452d
22/09/09 17:37:30 INFO [shutdown-hook-0] ShutdownHookManager: Deleting directory /tmp/spark-a8650bb9-cc9c-477e-9f94-91565b3de9ff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@linkis.apache.org
For additional commands, e-mail: notifications-help@linkis.apache.org