You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/08/02 13:47:08 UTC
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11265: [Bug] [Module Name] Using sqoop to export from hive to mysql reports an error
github-actions[bot] commented on issue #11265:
URL: https://github.com/apache/dolphinscheduler/issues/11265#issuecomment-1202605137
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
[LOG-PATH]: /opt/apache-dolphinscheduler-2.0.6-SNAPSHOT-bin/logs/6401667646944_3/49374/135527.log, [HOST]: 172.16.20.84
[
[INFO] 2022-08-02 20:39:51.658 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]] - tenantCode user:nddc, task dir:49374_135527
[INFO] 2022-08-02 20:39:51.658 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]] - create command file:/tmp/dolphinscheduler/exec/process/5690955182400/6401667646944_3/49374/135527/49374_135527.command
[INFO] 2022-08-02 20:39:51.658 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /opt/apache-dolphinscheduler-2.0.6-SNAPSHOT-bin/conf/env/dolphinscheduler_env.sh
sqoop export -D mapred.job.name=hive2mysql_dim_channel_tag_tree -m 1 --hcatalog-database nddc --hcatalog-table dim_channel_tag_tree --connect "jdbc:mysql://mysql57.rdsm82xiibwafef.rds.bj.baidubce.com:3306/x_data_ali?useUnicode=true&tinyInt1isBit=false&serverTimezone=Asia/Shanghai&characterEncoding=utf-8&autoReconnect=true&&allowLoadLocalInfile=false&autoDeserialize=false&allowLocalInfile=false&allowUrlInLocalInfile=false" --username user_2q33AUJxkQ --password "2q94TTMPO3" --table dim_channel_tag_tree --columns channel_code,channel_name,channel_order,tag_code_1,tag_name_1,tag_order_1,tag_code_2,tag_name_2,tag_order_2,tag_code_3,tag_name_3,tag_order_3,tag_code_4,tag_name_4,tag_order_4,tag_code_5,tag_name_5,tag_order_5,tag_code_6,tag_name_6,tag_order_6 --fields-terminated-by '\001'
[INFO] 2022-08-02 20:39:51.661 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]] - task run command: sudo -u nddc sh /tmp/dolphinscheduler/exec/process/5690955182400/6401667646944_3/49374/135527/49374_135527.command
[INFO] 2022-08-02 20:39:51.662 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]] - process start, process id is: 7931
[INFO] 2022-08-02 20:39:52.663 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]-getOutputLogService] - -> welcome to use bigdata scheduling system...
Warning: /opt/bmr/sqoop/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/bmr/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/bmr/sqoop/bin/../../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
[INFO] 2022-08-02 20:39:54.664 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]-getOutputLogService] - -> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/bmr/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/bmr/tez/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/bmr/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
22/08/02 20:39:54 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-SNAPSHOT
22/08/02 20:39:54 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
22/08/02 20:39:54 WARN tool.BaseSqoopTool: Output field/record delimiter options are not useful in HCatalog jobs for most of the output types except text based formats is text. It is better to use --hive-import in those cases. For non text formats,
22/08/02 20:39:54 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
22/08/02 20:39:54 INFO tool.CodeGenTool: Beginning code generation
[INFO] 2022-08-02 20:39:55.665 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]-getOutputLogService] - -> 22/08/02 20:39:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `dim_channel_tag_tree` AS t LIMIT 1
22/08/02 20:39:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `dim_channel_tag_tree` AS t LIMIT 1
22/08/02 20:39:55 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop
[INFO] 2022-08-02 20:40:00.665 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]-getOutputLogService] - -> Note: /tmp/sqoop-nddc/compile/da2ed2e77f28d966d366dcf638ab7d55/dim_channel_tag_tree.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
22/08/02 20:39:59 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-nddc/compile/da2ed2e77f28d966d366dcf638ab7d55/dim_channel_tag_tree.jar
22/08/02 20:39:59 INFO mapreduce.ExportJobBase: Beginning export of dim_channel_tag_tree
22/08/02 20:40:00 INFO mapreduce.ExportJobBase: Configuring HCatalog for export job
22/08/02 20:40:00 INFO hcat.SqoopHCatUtilities: Configuring HCatalog specific details for job
22/08/02 20:40:00 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `dim_channel_tag_tree` AS t LIMIT 1
22/08/02 20:40:00 INFO hcat.SqoopHCatUtilities: Database column names projected : [channel_code, channel_name, channel_order, tag_code_1, tag_name_1, tag_order_1, tag_code_2, tag_name_2, tag_order_2, tag_code_3, tag_name_3, tag_order_3, tag_code_4, tag_name_4, tag_order_4, tag_code_5, tag_name_5, tag_order_5, tag_code_6, tag_name_6, tag_order_6]
22/08/02 20:40:00 INFO hcat.SqoopHCatUtilities: Database column name - info map :
channel_name : [Type : 12,Precision : 100,Scale : 0]
tag_order_1 : [Type : 4,Precision : 11,Scale : 0]
tag_order_4 : [Type : 4,Precision : 11,Scale : 0]
tag_order_5 : [Type : 4,Precision : 11,Scale : 0]
tag_order_2 : [Type : 4,Precision : 11,Scale : 0]
tag_order_3 : [Type : 4,Precision : 11,Scale : 0]
channel_order : [Type : 4,Precision : 11,Scale : 0]
channel_code : [Type : 12,Precision : 20,Scale : 0]
tag_code_6 : [Type : 12,Precision : 50,Scale : 0]
tag_code_5 : [Type : 12,Precision : 50,Scale : 0]
tag_name_4 : [Type : 12,Precision : 200,Scale : 0]
tag_name_3 : [Type : 12,Precision : 200,Scale : 0]
tag_name_6 : [Type : 12,Precision : 200,Scale : 0]
tag_order_6 : [Type : 4,Precision : 11,Scale : 0]
tag_name_5 : [Type : 12,Precision : 200,Scale : 0]
tag_code_2 : [Type : 12,Precision : 50,Scale : 0]
tag_code_1 : [Type : 12,Precision : 50,Scale : 0]
tag_name_2 : [Type : 12,Precision : 200,Scale : 0]
tag_code_4 : [Type : 12,Precision : 50,Scale : 0]
tag_name_1 : [Type : 12,Precision : 200,Scale : 0]
tag_code_3 : [Type : 12,Precision : 50,Scale : 0]
22/08/02 20:40:00 INFO conf.HiveConf: Found configuration file file:/opt/bmr/hive/conf/hive-site.xml
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.metastore.local does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.enforce.sorting does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.materializedview.rewriting.incremental does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.server2.materializedviews.registry.impl does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.server2.webui.cors.allowed.headers does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.load.data.owner does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.service.metrics.codahale.reporter.classes does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.create.as.insert.only does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.mapred.supports.subdirectories does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.metastore.db.type does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.metastore.client.capability.check does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.tez.cartesian-product.enabled does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.metastore.warehouse.external.dir does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.enforce.bucketing does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.optimize.mapjoin.mapreduce does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.server2.webui.enable.cors does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.txn.strict.locking.mode does not exist
22/08/02 20:40:00 INFO hive.metastore: HMS client filtering is enabled.
22/08/02 20:40:00 INFO hive.metastore: Trying to connect to metastore with URI thrift://bmr-master-f9bd295-2:9083
[INFO] 2022-08-02 20:40:01.304 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]] - process has exited, execute path:/tmp/dolphinscheduler/exec/process/5690955182400/6401667646944_3/49374/135527, processId:7931 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1
[INFO] 2022-08-02 20:40:01.666 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]-getOutputLogService] - -> 22/08/02 20:40:00 INFO hive.metastore: Opened a connection to metastore, current connections: 1
22/08/02 20:40:00 INFO hive.metastore: Connected to metastore.
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.metastore.local does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.enforce.sorting does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.materializedview.rewriting.incremental does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.server2.materializedviews.registry.impl does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.server2.webui.cors.allowed.headers does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.load.data.owner does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.service.metrics.codahale.reporter.classes does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.create.as.insert.only does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.mapred.supports.subdirectories does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.metastore.db.type does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.metastore.client.capability.check does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.tez.cartesian-product.enabled does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.metastore.warehouse.external.dir does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.enforce.bucketing does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.optimize.mapjoin.mapreduce does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.server2.webui.enable.cors does not exist
22/08/02 20:40:00 WARN conf.HiveConf: HiveConf of name hive.txn.strict.locking.mode does not exist
22/08/02 20:40:01 INFO hcat.SqoopHCatUtilities: HCatalog full table schema fields = [channel_code, channel_name, channel_order, tag_code_1, tag_name_1, tag_order_1, tag_code_2, tag_name_2, tag_order_2, tag_code_3, tag_name_3, tag_order_3, tag_code_4, tag_name_4, tag_order_4, tag_code_5, tag_name_5, tag_order_5, tag_code_6, tag_name_6, tag_order_6, update_time]
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.metastore.local does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.enforce.sorting does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.materializedview.rewriting.incremental does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.server2.materializedviews.registry.impl does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.server2.webui.cors.allowed.headers does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.hook.proto.base-directory does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.load.data.owner does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.service.metrics.codahale.reporter.classes does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.strict.managed.tables does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.create.as.insert.only does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.mapred.supports.subdirectories does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.metastore.db.type does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.metastore.client.capability.check does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.tez.cartesian-product.enabled does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.metastore.warehouse.external.dir does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.enforce.bucketing does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.optimize.mapjoin.mapreduce does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.server2.webui.enable.cors does not exist
22/08/02 20:40:01 WARN conf.HiveConf: HiveConf of name hive.txn.strict.locking.mode does not exist
22/08/02 20:40:01 ERROR tool.ExportTool: Encountered IOException running export job: org.apache.hive.hcatalog.common.HCatException : 2001 : Error setting output information. Cause : org.apache.thrift.TApplicationException: Invalid method name: 'get_index_names'
22/08/02 20:40:01 INFO hive.metastore: Closed a connection to metastore, current connections: 0
[INFO] 2022-08-02 20:40:01.667 [TaskLogInfo- - [taskAppId=TASK-6401667646944_3-49374-135527]-getOutputLogService] - FINALIZE_SESSION
### What you expected to happen
sqoop export mysql success
### How to reproduce
### Anything else
_No response_
### Version
2.0.5
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org