You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2023/01/07 09:17:44 UTC
[GitHub] [incubator-seatunnel] xiaofan2022 opened a new issue, #3895: [Bug] [Sink] java.net.UnknownHostException
xiaofan2022 opened a new issue, #3895:
URL: https://github.com/apache/incubator-seatunnel/issues/3895
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Caused by: java.net.UnknownHostException: nameservice1
### SeaTunnel Version
2.3.0
### SeaTunnel Config
```conf
env {
execution.parallelism = 20
job.mode = "BATCH"
}
source {
Jdbc {
parallelism=20
url = "jdbc:mysql:host:3306/log"
driver = "com.mysql.cj.jdbc.Driver"
connection_check_timeout_sec = 100
user = ""
password = ""
query = "select * from table limit 100 "
partition_column = "id"
partition_num = 20
}
}
transform {
}
sink {
Hive {
table_name = "test.log"
parallelism=10
metastore_uri = "thrift://master1:9083"
sink_columns = ["id", "topic"]
}
}
```
### Running Command
```shell
sh ./bin/seatunnel.sh --config zeta_mysql_hive.conf
```
### Error Exception
```log
Caused by: java.net.UnknownHostException: nameservice1
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:445) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:132) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:351) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:285) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:168) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3237) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3286) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3254) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:478) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:462) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.1-SNAPSHOT]
at org.apache.seatunnel.connectors.seatunnel.file.sink.util.FileSystemUtils.getFileSystem(FileSystemUtils.java:67) ~[classes/:?]
at org.apache.seatunnel.connectors.seatunnel.file.sink.util.FileSystemUtils.getOutputStream(FileSystemUtils.java:73) ~[classes/:?]
at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.TextWriteStrategy.getOrCreateOutputStream(TextWriteStrategy.java:114) ~[classes/:?]
at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.TextWriteStrategy.write(TextWriteStrategy.java:75) ~[classes/:?]
at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSinkWriter.write(BaseFileSinkWriter.java:108) ~[classes/:?]
at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSinkWriter.write(BaseFileSinkWriter.java:43) ~[classes/:?]
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:161) ~[classes/:?]
... 15 more
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #3895: [Bug] [Sink] java.net.UnknownHostException
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer commented on issue #3895:
URL: https://github.com/apache/incubator-seatunnel/issues/3895#issuecomment-1425100572
> hdfs_site_path is supported in hive sink？
yup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] 18851652792 commented on issue #3895: [Bug] [Sink] java.net.UnknownHostException
Posted by "18851652792 (via GitHub)" <gi...@apache.org>.
18851652792 commented on issue #3895:
URL: https://github.com/apache/incubator-seatunnel/issues/3895#issuecomment-1486461812
I also appears the question, can demonstrate how to configure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Sink] java.net.UnknownHostException [seatunnel]
Posted by "taodejin (via GitHub)" <gi...@apache.org>.
taodejin commented on issue #3895:
URL: https://github.com/apache/seatunnel/issues/3895#issuecomment-1864089716
env {
# You can set flink configuration here
parallelism = 3
job.name="test_hive_source_to_hive"
}
source {
Hive {
table_name = "default.flink_test"
metastore_uri = "thrift://10.251.181.13:9083"
}
}
sink {
# choose stdout output plugin to output data to console
Hive {
table_name = "default.datax_test"
metastore_uri = "thrift://10.251.181.13:9083"
}
}
I also encountered this problem, how to solve it？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer commented on issue #3895: [Bug] [Sink] java.net.UnknownHostException
Posted by GitBox <gi...@apache.org>.
TyrantLucifer commented on issue #3895:
URL: https://github.com/apache/incubator-seatunnel/issues/3895#issuecomment-1374804572
Add `hdfs_site_path` configuration in hive plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3895: [Bug] [Sink] java.net.UnknownHostException
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3895:
URL: https://github.com/apache/incubator-seatunnel/issues/3895#issuecomment-1465345085
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3895: [Bug] [Sink] java.net.UnknownHostException
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3895:
URL: https://github.com/apache/incubator-seatunnel/issues/3895#issuecomment-1425013898
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] TyrantLucifer closed issue #3895: [Bug] [Sink] java.net.UnknownHostException
Posted by "TyrantLucifer (via GitHub)" <gi...@apache.org>.
TyrantLucifer closed issue #3895: [Bug] [Sink] java.net.UnknownHostException
URL: https://github.com/apache/incubator-seatunnel/issues/3895
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xiaofan2022 commented on issue #3895: [Bug] [Sink] java.net.UnknownHostException
Posted by GitBox <gi...@apache.org>.
xiaofan2022 commented on issue #3895:
URL: https://github.com/apache/incubator-seatunnel/issues/3895#issuecomment-1376831763
hdfs_site_path is supported in hive sink？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] LIXINING0518 commented on issue #3895: [Bug] [Sink] java.net.UnknownHostException
Posted by "LIXINING0518 (via GitHub)" <gi...@apache.org>.
LIXINING0518 commented on issue #3895:
URL: https://github.com/apache/incubator-seatunnel/issues/3895#issuecomment-1486484335
> 我也出现了这个问题，你能提供配置hdfs-site.xml吗？
env {
job.mode = "BATCH"
#execution.checkpoint.interval = 10000
#execution.checkpoint.data-uri = "hdfs://hdfscluster:9000/checkpoint"
}
source {
Jdbc {
url = "jdbc:mysql://10.3.7.226/wisdom_microhabitat?serverTimezone=GMT%2b8"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "123456"
query = "select * from microhabitat_total_2022"
}
}
transform {
}
sink {
Hive {
table_name = "default.microhabitat_total_2022"
metastore_uri = "thrift://10.3.6.24:9083"
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org