You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "e-mhui (via GitHub)" <gi...@apache.org> on 2025/01/11 13:06:11 UTC
[I] [Bug] [Sftp Reader] If there are wildcard file names in the sftp directory and there happen to be other matching file names, a uniqueness error will occur [seatunnel]
e-mhui opened a new issue, #8500:
URL: https://github.com/apache/seatunnel/issues/8500
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
If there are wildcard file names in the sftp directory and there happen to be other matching file names, a uniqueness error will occur.
For example, there are two files, `a * b.txt` and `abc.txt`, in the `/data` directory. When configuring/data/a * b.txt, a uniqueness error occurs.
<img width="1421" alt="Clipboard_Screenshot_1736600413" src="https://github.com/user-attachments/assets/8a4acb91-9c46-4e76-ac47-82cb7af4b197" />
### SeaTunnel Version
dev
### SeaTunnel Config
```conf
env {
parallelism = 1
job.mode = "BATCH"
}
source {
SftpFile {
host = ""
port = xxx
user = xxx
password = xxx
path = "/data/a*b.txt"
file_format_type = "text"
field_delimiter = ","
schema = {
fields {
c_int = int
c_string = string
}
}
}
}
transform {
}
sink {
console {
}
}
```
### Running Command
```shell
`org.apache.seatunnel.example.engine.SeaTunnelEngineLocalExample`
public static void main(String[] args)
throws FileNotFoundException, URISyntaxException, CommandException {
String configurePath = args.length > 0 ? args[0] : "/examples/sftp_to_console.conf";
String configFile = getTestConfigFile(configurePath);
ClientCommandArgs clientCommandArgs = new ClientCommandArgs();
clientCommandArgs.setConfigFile(configFile);
clientCommandArgs.setCheckConfig(false);
clientCommandArgs.setJobName(Paths.get(configFile).getFileName().toString());
// Change Execution Mode to CLUSTER to use client mode, before do this, you should start
// SeaTunnelEngineClusterServerExample
clientCommandArgs.setMasterType(MasterType.LOCAL);
SeaTunnel.run(clientCommandArgs.buildCommand());
}
```
### Error Exception
```log
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException: ErrorCode:[FILE-08], ErrorDescription:[File read failed] - Read data from this file [default.default.default_sftp://xxxx/data/aa*bbb.txt] failed
at org.apache.seatunnel.connectors.seatunnel.file.source.reader.MultipleTableFileSourceReader.pollNext(MultipleTableFileSourceReader.java:85)
at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:159)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:169)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:694)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1019)
at org.apache.seatunnel.api.tracing.MDCRunnable.run(MDCRunnable.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: 4: /data/aa*bbb.txt is not unique: [/data/aa123bbb.txt, /data/aa*bbb.txt]
at org.apache.seatunnel.connectors.seatunnel.file.sftp.system.SFTPFileSystem.open(SFTPFileSystem.java:495)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.lambda$getInputStream$11(HadoopFileSystemProxy.java:195)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.execute(HadoopFileSystemProxy.java:327)
at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.getInputStream(HadoopFileSystemProxy.java:195)
at org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.resolveArchiveCompressedInputStream(AbstractReadStrategy.java:272)
at org.apache.seatunnel.connectors.seatunnel.file.source.reader.TextReadStrategy.read(TextReadStrategy.java:71)
at org.apache.seatunnel.connectors.seatunnel.file.source.reader.MultipleTableFileSourceReader.pollNext(MultipleTableFileSourceReader.java:81)
... 12 more
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Sftp Reader] If there are wildcard file names in the sftp directory and there happen to be other matching file names, a uniqueness error will occur [seatunnel]
Posted by "hailin0 (via GitHub)" <gi...@apache.org>.
hailin0 closed issue #8500: [Bug] [Sftp Reader] If there are wildcard file names in the sftp directory and there happen to be other matching file names, a uniqueness error will occur
URL: https://github.com/apache/seatunnel/issues/8500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org