You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "justlkp (via GitHub)" <gi...@apache.org> on 2025/01/10 03:40:14 UTC
[I] [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed [seatunnel]
justlkp opened a new issue, #8493:
URL: https://github.com/apache/seatunnel/issues/8493
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Within the same network segment, the FTP connector can normally read files. However, when accessing the same FTP service with an IP address that is not in the same segment, access is not possible. Yet, under the same circumstances, using an FTP client such as FileZilla allows for normal access. After multiple tests in this scenario, it was found that the FTP connector only functions properly when used within the same network segment.
To clarify further:
When both the FTP server and the connector are on the same subnet, file reading works as expected.
When the FTP server and a device using the connector are on different subnets, the connection fails.
An FTP client like FileZilla can successfully connect to the FTP server regardless of being on the same or a different subnet, indicating that the issue is specific to the FTP connector's configuration or limitations.
在同一个网段下,可以使用ftp连接器可以正常读取文件,但是访问同一个ftp服务,ip地址不在同一个网段,则无法获取ftp文件列表,显示文件不存在,显示Get file list from this path [/test/test.csv] failed , 但是同样情况下使用ftp客户端,比如filezilla可以正常访问,多次测试此种情况,发现只有在同一个网段下该ftp连接器才可以正常使用,已经使用vsftpd 3.0.2和vsftpd 3.0.3多次测试,复现这个问题
successful config
```sh
env {
parallelism = 1
job.mode = "BATCH"
}
source {
FtpFile {
result_table_name = "fake2"
host = "192.168.8.44"
port = 19872
user = "test"
password = "123456"
path = "/test/test.csv"
file_filter_pattern = "test.csv"
file_format_type = "csv"
schema = {
fields {
name = "string"
age = "string"
}
}
}
}
transform {
}
sink {
Jdbc {
url = "jdbc:mysql://192.168.8.42:3306/test?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=false&serverTimezone=GMT%2B8"
driver = "com.mysql.cj.jdbc.Driver"
user = "test"
password = "123456"
database = "sea_test"
table = "dev_user_ftp"
schema_save_mode = "RECREATE_SCHEMA"
data_save_mode="APPEND_DATA"
generate_sink_sql = true
}
}
```
fail config
```sh
env {
parallelism = 1
job.mode = "BATCH"
}
source {
FtpFile {
result_table_name = "fake2"
host = "111.xxx.xx.xxx"
port = 19872
user = "test"
password = "123456"
path = "/test/test.csv"
file_filter_pattern = "test.csv"
file_format_type = "csv"
schema = {
fields {
name = "string"
age = "string"
}
}
}
}
transform {
}
sink {
Jdbc {
url = "jdbc:mysql://192.168.8.42:3306/test?useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=false&serverTimezone=GMT%2B8"
driver = "com.mysql.cj.jdbc.Driver"
user = "test"
password = "123456"
database = "sea_test"
table = "dev_user_ftp"
schema_save_mode = "RECREATE_SCHEMA"
data_save_mode="APPEND_DATA"
generate_sink_sql = true
}
}
```
### SeaTunnel Version
2.3.8
### SeaTunnel Config
```conf
seatunnel:
engine:
http:
enable-http: true
port: 6086
history-job-expire-minutes: 1440
backup-count: 1
queue-type: blockingqueue
print-execution-info-interval: 60
print-job-metrics-info-interval: 60
slot-service:
dynamic-slot: true
checkpoint:
interval: 300000
timeout: 60000
storage:
type: hdfs
max-retained: 3
plugin-config:
namespace: /tmp/seatunnel/checkpoint_snapshot
storage.type: hdfs
fs.defaultFS: file:///tmp/ # Ensure that the directory has written permission
telemetry:
metric:
enabled: false
```
### Running Command
```shell
./bin/seatunnel.sh --config ./config/testftp2mysqlremote.template -m local
```
### Error Exception
```log
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:213)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.example.engine.SeaTunnelEngineExample.main(SeaTunnelEngineExample.java:43)
Caused by: org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException: ErrorCode:[FILE-03], ErrorDescription:[Get file list failed] - Get file list from this path [/test/test.csv] failed
at org.apache.seatunnel.connectors.seatunnel.file.ftp.source.FtpFileSource.prepare(FtpFileSource.java:96)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:83)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:370)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:209)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:114)
at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:182)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:160)
... 2 more
Caused by: java.io.FileNotFoundException: File /test/test.csv does not exist.
at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.getFileStatus(SeaTunnelFTPFileSystem.java:523)
at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.listStatus(SeaTunnelFTPFileSystem.java:439)
at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.listStatus(SeaTunnelFTPFileSystem.java:421)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.listStatus(HadoopFileSystemProxy.java:154)
at org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.getFileNamesByPath(AbstractReadStrategy.java:110)
at org.apache.seatunnel.connectors.seatunnel.file.ftp.source.FtpFileSource.prepare(FtpFileSource.java:92)
... 8 more
Process finished with exit code 1
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
Java 1.8.0_202
### Screenshots

### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8493:
URL: https://github.com/apache/seatunnel/issues/8493#issuecomment-3092661386
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed [seatunnel]
Posted by "liuxiaobopro (via GitHub)" <gi...@apache.org>.
liuxiaobopro commented on issue #8493:
URL: https://github.com/apache/seatunnel/issues/8493#issuecomment-4002839448
> 2.3.12遇到相同问题, 切换被动模式 passive_local 也不行
{
"env": {
"job.name": "ftp_ftp_938cba09-a58f-4d72-a1f5-2839d7b6a946",
"job.mode": "BATCH",
"parallelism": 1
},
"source": [
{
"plugin_name": "FtpFile",
"plugin_output": "/1111.txt",
"host": "127.0.0.1",
"port": 21,
"user": "ftpuser",
"password": "ftppass",
"path": "/1111.txt",
"file_format_type": "binary",
"connection_mode": "passive_local"
}
],
"sink": [
{
"plugin_name": "FtpFile",
"host": "127.0.0.1",
"port": 21,
"user": "ftpuser",
"password": "ftppass",
"path": "20260305151738",
"connection_mode": "passive_local",
"file_format_type": "binary"
}
]
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8493: [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed
URL: https://github.com/apache/seatunnel/issues/8493
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8493:
URL: https://github.com/apache/seatunnel/issues/8493#issuecomment-3064423737
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed [seatunnel]
Posted by "FakeTrader (via GitHub)" <gi...@apache.org>.
FakeTrader commented on issue #8493:
URL: https://github.com/apache/seatunnel/issues/8493#issuecomment-2723841803
resolved by switching to passive mode.
```java
client.enterLocalPassiveMode();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed [seatunnel]
Posted by "liuxiaobopro (via GitHub)" <gi...@apache.org>.
liuxiaobopro commented on issue #8493:
URL: https://github.com/apache/seatunnel/issues/8493#issuecomment-4002800393
2.3.12遇到相同问题, 切换被动模式 passive_local 也不行
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed [seatunnel]
Posted by "FakeTrader (via GitHub)" <gi...@apache.org>.
FakeTrader commented on issue #8493:
URL: https://github.com/apache/seatunnel/issues/8493#issuecomment-2720589482
same issue here.
i'm running an SFTPGo server in Docker, and the FTP server rejected the request with "501 Your request does not meet the configured security requirements" reply. log showed the connection IP was 192.168.65.1 which belongs to the Docker subnet.
using FTPClient directly or other FTP GUI client works fine.
more details https://github.com/apache/seatunnel/issues/8202#issuecomment-2720202542.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed [seatunnel]
Posted by "kaantecik (via GitHub)" <gi...@apache.org>.
kaantecik commented on issue #8493:
URL: https://github.com/apache/seatunnel/issues/8493#issuecomment-2639918489
facing with same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [connector-file-ftp] When accessing an FTP server in a different subnet, it results in a failure to retrieve the FTP directory listing,show: Get file list failed [seatunnel]
Posted by "mesutmutluitu (via GitHub)" <gi...@apache.org>.
mesutmutluitu commented on issue #8493:
URL: https://github.com/apache/seatunnel/issues/8493#issuecomment-2663037138
same problem here
`POST]{uri='/hazelcast/rest/maps/submit-job'}AbstractTextCommand[HTTP_POST]{requestId=2}
org.apache.seatunnel.api.table.factory.FactoryException: ErrorCode:[API-06], ErrorDescription:[Factory initialize failed] - Unable to create a source for identifier 'FtpFile'.
at org.apache.seatunnel.api.table.factory.FactoryUtil.restoreAndPrepareSource(FactoryUtil.java:119) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:68) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:404) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:236) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.rest.RestJobExecutionEnvironment.getLogicalDag(RestJobExecutionEnvironment.java:86) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.rest.RestJobExecutionEnvironment.build(RestJobExecutionEnvironment.java:122) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.rest.service.BaseService.submitJobInternal(BaseService.java:505) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.rest.service.JobInfoService.submitJob(JobInfoService.java:173) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handleSubmitJob(RestHttpPostCommandProcessor.java:114) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handle(RestHttpPostCommandProcessor.java:82) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handle(RestHttpPostCommandProcessor.java:44) ~[seatunnel-starter.jar:2.3.9]
at com.hazelcast.internal.ascii.TextCommandServiceImpl$CommandExecutor.run(TextCommandServiceImpl.java:402) ~[seatunnel-starter.jar:2.3.9]
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[seatunnel-starter.jar:2.3.9]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[seatunnel-starter.jar:2.3.9]
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[seatunnel-starter.jar:2.3.9]
Caused by: org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException: ErrorCode:[FILE-03], ErrorDescription:[Get file list failed] - Get file list from this path [/dir2] failed
at org.apache.seatunnel.connectors.seatunnel.file.config.BaseFileSourceConfig.parseFilePaths(BaseFileSourceConfig.java:76) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.config.BaseFileSourceConfig.<init>(BaseFileSourceConfig.java:64) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.config.FTPFileSourceConfig.<init>(FTPFileSourceConfig.java:43) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.config.MultipleTableFTPFileSourceConfig.getBaseSourceConfig(MultipleTableFTPFileSourceConfig.java:32) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.config.BaseMultipleTableFileSourceConfig.parseFromFileSourceConfig(BaseMultipleTableFileSourceConfig.java:56) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.config.BaseMultipleTableFileSourceConfig.<init>(BaseMultipleTableFileSourceConfig.java:41) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.config.MultipleTableFTPFileSourceConfig.<init>(MultipleTableFTPFileSourceConfig.java:27) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.source.FtpFileSource.<init>(FtpFileSource.java:27) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.source.FtpFileSourceFactory.lambda$createSource$0(FtpFileSourceFactory.java:48) ~[?:?]
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:131) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.api.table.factory.FactoryUtil.restoreAndPrepareSource(FactoryUtil.java:91) ~[seatunnel-starter.jar:2.3.9]
... 17 more
Caused by: java.io.FileNotFoundException: File /dir2 does not exist.
at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.getFileStatus(SeaTunnelFTPFileSystem.java:492) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.listStatus(SeaTunnelFTPFileSystem.java:439) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.listStatus(SeaTunnelFTPFileSystem.java:424) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.lambda$listStatus$8(HadoopFileSystemProxy.java:183) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.execute(HadoopFileSystemProxy.java:327) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.listStatus(HadoopFileSystemProxy.java:183) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.getFileNamesByPath(AbstractReadStrategy.java:110) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.config.BaseFileSourceConfig.parseFilePaths(BaseFileSourceConfig.java:73) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.config.BaseFileSourceConfig.<init>(BaseFileSourceConfig.java:64) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.config.FTPFileSourceConfig.<init>(FTPFileSourceConfig.java:43) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.config.MultipleTableFTPFileSourceConfig.getBaseSourceConfig(MultipleTableFTPFileSourceConfig.java:32) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.config.BaseMultipleTableFileSourceConfig.parseFromFileSourceConfig(BaseMultipleTableFileSourceConfig.java:56) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.config.BaseMultipleTableFileSourceConfig.<init>(BaseMultipleTableFileSourceConfig.java:41) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.config.MultipleTableFTPFileSourceConfig.<init>(MultipleTableFTPFileSourceConfig.java:27) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.source.FtpFileSource.<init>(FtpFileSource.java:27) ~[?:?]
at org.apache.seatunnel.connectors.seatunnel.file.ftp.source.FtpFileSourceFactory.lambda$createSource$0(FtpFileSourceFactory.java:48) ~[?:?]
at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareSource(FactoryUtil.java:131) ~[seatunnel-starter.jar:2.3.9]
at org.apache.seatunnel.api.table.factory.FactoryUtil.restoreAndPrepareSource(FactoryUtil.java:91) ~[seatunnel-starter.jar:2.3.9]
... 17 more`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org