You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Chris Wolf <cw...@gmail.com> on 2013/04/17 17:39:53 UTC
Why doesn't SFTP "move" option work??
Trying to use Sftp component. I am using localWorkDirectory because
I don't want the whole files loaded into memory.
Seems like the files *are* being downloaded but they are NOT being
moved, on the remote side to the directory indicated by the "move"
option.
Here, on a poll, it gets a file, while downloading to
"localWorkDirectory", the file's name is appended with ".inprogress",
then it's
renamed to the base file name without ".inprogress" - so far, so good,
except on the server-side, it's not moved to the "move" directory.
[#4 - sftp://localhost/download] SftpOperations DEBUG Retrieve file to
local work file result: true
[#4 - sftp://localhost/download] SftpOperations TRACE Renaming local
in progress file from:
\tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv.inprogress to:
\tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv
[#4 - sftp://localhost/download] SftpConsumer TRACE Retrieved file:
download/ADPT5_MKT_20120319_VOIR_DAILY.csv from:
Endpoint[sftp://localhost/download]
...then, at poll n+1, I see:
[#4 - sftp://localhost/download] SftpConsumer TRACE
SftpFile[fileName=ADPT5_MKT_20120319_VOIR_DAILY.csv,
longName=-rwxrwxrwx 1 adpt5 adpt5 298313 Apr 17 09:42
ADPT5_MKT_20120319_VOIR_DAILY.csv, dir=false]
[#4 - sftp://localhost/download] SftpConsumer TRACE Skipping as file
is already in progress: ADPT5_MKT_20120319_VOIR_DAILY.csv
I guess that's the idempotent mechanism, but that file should have
been moved out of that directory.
SftpConfiguration:
chmod = null,
ciphers = null,
compression = 0,
connectTimeout = 10000,
directory = download,
host = localhost,
knownHostsFile = null,
password = adpt5,
port = 22,
privateKeyFile = null,
privateKeyFilePassphrase = null,
protocol = sftp,
separator = UNIX,
serverAliveCountMax = 1,
serverAliveInterval = 0,
siteCommand = null,
soTimeout = 0,
strictHostKeyChecking = no,
timeout = 30000,
username = adpt5,
binary = true,
passiveMode = false,
stepwise = true,
throwExceptionOnConnectFailed = false
[1) thread #0 - timer://kickoff] INFO SftpEndpoint:
antFilter = null,
bufferSize = 131072,
camelContext = CamelContext(camel-1),
charset = null,
component = org.apache.camel.component.file.remote.SftpComponent@1e34f445,
configuration = download,
consumerProperties = null,
doneFileName = null,
endpointConfiguration =
org.apache.camel.impl.MappedEndpointConfiguration@8292f75a,
endpointKey = sftp://localhost/download,
endpointUri = sftp://localhost/download,
exchangePattern = InOnly,
exchanges = [Exchange[ADPT5_MKT_20090211_MICF.csv],
Exchange[ADPT5_MKT_20090211_MINOCF.csv],
Exchange[ADPT5_MKT_20090305_VOE.csv],
Exchange[ADPT5_MKT_20090305_VOIR.csv],
Exchange[ADPT5_MKT_20120319_DAILY.csv],
Exchange[ADPT5_MKT_20120319_FX.csv],
Exchange[ADPT5_MKT_20120319_MTX_DAILY.csv],
Exchange[ADPT5_MKT_20120319_VOIR_DAILY.csv],
Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-D.csv],
Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-M.csv],
Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-Y.csv]],
exclude = null,
exclusiveReadLockStrategy = null,
fileExist = Override,
fileName = null,
fileSeparator = /,
filter = ms.algo.adapt.camel.cpmd.CPMDFileFilter@54aa2db,
genericFileProcessStrategy =
org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530,
id = endpoint13,
idempotentRepository = null,
include = null,
inProgressRepository =
org.apache.camel.processor.idempotent.MemoryIdempotentRepository@563b100c,
localWorkDirectory = /tmp/local/ftpwork,
maxDepth = 2147483647,
maximumReconnectAttempts = 3,
maxMessagesPerPoll = 0,
minDepth = 0,
move = ${file:parent}/../archive/${file:onlyname},,
moveExisting = null,
moveFailed = null,
preMove = null,
processStrategy =
org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530,
readLock = none,
readLockCheckInterval = 5000,
readLockMinLength = 1,
readLockTimeout = 20000,
reconnectDelay = 1000,
scheme = sftp,
sortBy = null,
sorter = null,
status = Started,
tempFileName = null,
tempPrefix = null,
version = 2.10.4,
allowNullBody = false,
autoCreate = true,
delete = false,
directoryMustExist = false,
disconnect = false,
eagerDeleteTargetFile = true,
eagerMaxMessagesPerPoll = true,
fastExistsCheck = false,
flatten = false,
keepLastModified = false,
lenientProperties = false,
noop = false,
recursive = false,
runAllowed = true,
singleton = true,
started = true,
starting = false,
startingDirectoryMustExist = false,
stopped = false,
stopping = false,
suspended = false,
suspending = false,
synchronous = false
Re: Why doesn't SFTP "move" option work??
Posted by Chris Wolf <cw...@gmail.com>.
I took the FromFtpMoveFileTest and modified it for SFTP - that works,
so "move" works
for SFTP. My problem is that because the current release requires
hard-coding the
URI for (s)ftp(s) consumer - I implemented a custom processor that
instantiates the SFTP component,
SFTP endpoint and SFTP consumer, all programatically. Obviously, I'm
missing something.
Unfortunately, it's pretty much a show-stopper for us to not be able
to SFTP *from* (consume)
dynamically (runtime-calculated) URIs.
On Wed, Apr 17, 2013 at 4:54 PM, Chris Wolf <cw...@gmail.com> wrote:
> I notice there's a unit test, FromFtpMoveFileTest, for plain FTP, but
> not for SFTP - should I file a bug, or am I doing something
> wrong?
>
> On Wed, Apr 17, 2013 at 11:39 AM, Chris Wolf <cw...@gmail.com> wrote:
>> Trying to use Sftp component. I am using localWorkDirectory because
>> I don't want the whole files loaded into memory.
>> Seems like the files *are* being downloaded but they are NOT being
>> moved, on the remote side to the directory indicated by the "move"
>> option.
>>
>>
>> Here, on a poll, it gets a file, while downloading to
>> "localWorkDirectory", the file's name is appended with ".inprogress",
>> then it's
>> renamed to the base file name without ".inprogress" - so far, so good,
>> except on the server-side, it's not moved to the "move" directory.
>>
>> [#4 - sftp://localhost/download] SftpOperations DEBUG Retrieve file to
>> local work file result: true
>> [#4 - sftp://localhost/download] SftpOperations TRACE Renaming local
>> in progress file from:
>> \tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv.inprogress to:
>> \tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv
>> [#4 - sftp://localhost/download] SftpConsumer TRACE Retrieved file:
>> download/ADPT5_MKT_20120319_VOIR_DAILY.csv from:
>> Endpoint[sftp://localhost/download]
>>
>> ...then, at poll n+1, I see:
>>
>> [#4 - sftp://localhost/download] SftpConsumer TRACE
>> SftpFile[fileName=ADPT5_MKT_20120319_VOIR_DAILY.csv,
>> longName=-rwxrwxrwx 1 adpt5 adpt5 298313 Apr 17 09:42
>> ADPT5_MKT_20120319_VOIR_DAILY.csv, dir=false]
>> [#4 - sftp://localhost/download] SftpConsumer TRACE Skipping as file
>> is already in progress: ADPT5_MKT_20120319_VOIR_DAILY.csv
>>
>> I guess that's the idempotent mechanism, but that file should have
>> been moved out of that directory.
>>
>>
>> SftpConfiguration:
>>
>> chmod = null,
>> ciphers = null,
>> compression = 0,
>> connectTimeout = 10000,
>> directory = download,
>> host = localhost,
>> knownHostsFile = null,
>> password = adpt5,
>> port = 22,
>> privateKeyFile = null,
>> privateKeyFilePassphrase = null,
>> protocol = sftp,
>> separator = UNIX,
>> serverAliveCountMax = 1,
>> serverAliveInterval = 0,
>> siteCommand = null,
>> soTimeout = 0,
>> strictHostKeyChecking = no,
>> timeout = 30000,
>> username = adpt5,
>> binary = true,
>> passiveMode = false,
>> stepwise = true,
>> throwExceptionOnConnectFailed = false
>>
>> [1) thread #0 - timer://kickoff] INFO SftpEndpoint:
>> antFilter = null,
>> bufferSize = 131072,
>> camelContext = CamelContext(camel-1),
>> charset = null,
>> component = org.apache.camel.component.file.remote.SftpComponent@1e34f445,
>> configuration = download,
>> consumerProperties = null,
>> doneFileName = null,
>> endpointConfiguration =
>> org.apache.camel.impl.MappedEndpointConfiguration@8292f75a,
>> endpointKey = sftp://localhost/download,
>> endpointUri = sftp://localhost/download,
>> exchangePattern = InOnly,
>> exchanges = [Exchange[ADPT5_MKT_20090211_MICF.csv],
>> Exchange[ADPT5_MKT_20090211_MINOCF.csv],
>> Exchange[ADPT5_MKT_20090305_VOE.csv],
>> Exchange[ADPT5_MKT_20090305_VOIR.csv],
>> Exchange[ADPT5_MKT_20120319_DAILY.csv],
>> Exchange[ADPT5_MKT_20120319_FX.csv],
>> Exchange[ADPT5_MKT_20120319_MTX_DAILY.csv],
>> Exchange[ADPT5_MKT_20120319_VOIR_DAILY.csv],
>> Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-D.csv],
>> Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-M.csv],
>> Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-Y.csv]],
>> exclude = null,
>> exclusiveReadLockStrategy = null,
>> fileExist = Override,
>> fileName = null,
>> fileSeparator = /,
>> filter = ms.algo.adapt.camel.cpmd.CPMDFileFilter@54aa2db,
>> genericFileProcessStrategy =
>> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530,
>> id = endpoint13,
>> idempotentRepository = null,
>> include = null,
>> inProgressRepository =
>> org.apache.camel.processor.idempotent.MemoryIdempotentRepository@563b100c,
>> localWorkDirectory = /tmp/local/ftpwork,
>> maxDepth = 2147483647,
>> maximumReconnectAttempts = 3,
>> maxMessagesPerPoll = 0,
>> minDepth = 0,
>> move = ${file:parent}/../archive/${file:onlyname},,
>> moveExisting = null,
>> moveFailed = null,
>> preMove = null,
>> processStrategy =
>> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530,
>> readLock = none,
>> readLockCheckInterval = 5000,
>> readLockMinLength = 1,
>> readLockTimeout = 20000,
>> reconnectDelay = 1000,
>> scheme = sftp,
>> sortBy = null,
>> sorter = null,
>> status = Started,
>> tempFileName = null,
>> tempPrefix = null,
>> version = 2.10.4,
>> allowNullBody = false,
>> autoCreate = true,
>> delete = false,
>> directoryMustExist = false,
>> disconnect = false,
>> eagerDeleteTargetFile = true,
>> eagerMaxMessagesPerPoll = true,
>> fastExistsCheck = false,
>> flatten = false,
>> keepLastModified = false,
>> lenientProperties = false,
>> noop = false,
>> recursive = false,
>> runAllowed = true,
>> singleton = true,
>> started = true,
>> starting = false,
>> startingDirectoryMustExist = false,
>> stopped = false,
>> stopping = false,
>> suspended = false,
>> suspending = false,
>> synchronous = false
Re: Why doesn't SFTP "move" option work??
Posted by Chris Wolf <cw...@gmail.com>.
I notice there's a unit test, FromFtpMoveFileTest, for plain FTP, but
not for SFTP - should I file a bug, or am I doing something
wrong?
On Wed, Apr 17, 2013 at 11:39 AM, Chris Wolf <cw...@gmail.com> wrote:
> Trying to use Sftp component. I am using localWorkDirectory because
> I don't want the whole files loaded into memory.
> Seems like the files *are* being downloaded but they are NOT being
> moved, on the remote side to the directory indicated by the "move"
> option.
>
>
> Here, on a poll, it gets a file, while downloading to
> "localWorkDirectory", the file's name is appended with ".inprogress",
> then it's
> renamed to the base file name without ".inprogress" - so far, so good,
> except on the server-side, it's not moved to the "move" directory.
>
> [#4 - sftp://localhost/download] SftpOperations DEBUG Retrieve file to
> local work file result: true
> [#4 - sftp://localhost/download] SftpOperations TRACE Renaming local
> in progress file from:
> \tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv.inprogress to:
> \tmp\local\ftpwork\ADPT5_MKT_20120319_VOIR_DAILY.csv
> [#4 - sftp://localhost/download] SftpConsumer TRACE Retrieved file:
> download/ADPT5_MKT_20120319_VOIR_DAILY.csv from:
> Endpoint[sftp://localhost/download]
>
> ...then, at poll n+1, I see:
>
> [#4 - sftp://localhost/download] SftpConsumer TRACE
> SftpFile[fileName=ADPT5_MKT_20120319_VOIR_DAILY.csv,
> longName=-rwxrwxrwx 1 adpt5 adpt5 298313 Apr 17 09:42
> ADPT5_MKT_20120319_VOIR_DAILY.csv, dir=false]
> [#4 - sftp://localhost/download] SftpConsumer TRACE Skipping as file
> is already in progress: ADPT5_MKT_20120319_VOIR_DAILY.csv
>
> I guess that's the idempotent mechanism, but that file should have
> been moved out of that directory.
>
>
> SftpConfiguration:
>
> chmod = null,
> ciphers = null,
> compression = 0,
> connectTimeout = 10000,
> directory = download,
> host = localhost,
> knownHostsFile = null,
> password = adpt5,
> port = 22,
> privateKeyFile = null,
> privateKeyFilePassphrase = null,
> protocol = sftp,
> separator = UNIX,
> serverAliveCountMax = 1,
> serverAliveInterval = 0,
> siteCommand = null,
> soTimeout = 0,
> strictHostKeyChecking = no,
> timeout = 30000,
> username = adpt5,
> binary = true,
> passiveMode = false,
> stepwise = true,
> throwExceptionOnConnectFailed = false
>
> [1) thread #0 - timer://kickoff] INFO SftpEndpoint:
> antFilter = null,
> bufferSize = 131072,
> camelContext = CamelContext(camel-1),
> charset = null,
> component = org.apache.camel.component.file.remote.SftpComponent@1e34f445,
> configuration = download,
> consumerProperties = null,
> doneFileName = null,
> endpointConfiguration =
> org.apache.camel.impl.MappedEndpointConfiguration@8292f75a,
> endpointKey = sftp://localhost/download,
> endpointUri = sftp://localhost/download,
> exchangePattern = InOnly,
> exchanges = [Exchange[ADPT5_MKT_20090211_MICF.csv],
> Exchange[ADPT5_MKT_20090211_MINOCF.csv],
> Exchange[ADPT5_MKT_20090305_VOE.csv],
> Exchange[ADPT5_MKT_20090305_VOIR.csv],
> Exchange[ADPT5_MKT_20120319_DAILY.csv],
> Exchange[ADPT5_MKT_20120319_FX.csv],
> Exchange[ADPT5_MKT_20120319_MTX_DAILY.csv],
> Exchange[ADPT5_MKT_20120319_VOIR_DAILY.csv],
> Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-D.csv],
> Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-M.csv],
> Exchange[ADPT5_MKT_20121121_HISTORY_EUR_IRBank-FZ-Y.csv]],
> exclude = null,
> exclusiveReadLockStrategy = null,
> fileExist = Override,
> fileName = null,
> fileSeparator = /,
> filter = ms.algo.adapt.camel.cpmd.CPMDFileFilter@54aa2db,
> genericFileProcessStrategy =
> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530,
> id = endpoint13,
> idempotentRepository = null,
> include = null,
> inProgressRepository =
> org.apache.camel.processor.idempotent.MemoryIdempotentRepository@563b100c,
> localWorkDirectory = /tmp/local/ftpwork,
> maxDepth = 2147483647,
> maximumReconnectAttempts = 3,
> maxMessagesPerPoll = 0,
> minDepth = 0,
> move = ${file:parent}/../archive/${file:onlyname},,
> moveExisting = null,
> moveFailed = null,
> preMove = null,
> processStrategy =
> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@67b14530,
> readLock = none,
> readLockCheckInterval = 5000,
> readLockMinLength = 1,
> readLockTimeout = 20000,
> reconnectDelay = 1000,
> scheme = sftp,
> sortBy = null,
> sorter = null,
> status = Started,
> tempFileName = null,
> tempPrefix = null,
> version = 2.10.4,
> allowNullBody = false,
> autoCreate = true,
> delete = false,
> directoryMustExist = false,
> disconnect = false,
> eagerDeleteTargetFile = true,
> eagerMaxMessagesPerPoll = true,
> fastExistsCheck = false,
> flatten = false,
> keepLastModified = false,
> lenientProperties = false,
> noop = false,
> recursive = false,
> runAllowed = true,
> singleton = true,
> started = true,
> starting = false,
> startingDirectoryMustExist = false,
> stopped = false,
> stopping = false,
> suspended = false,
> suspending = false,
> synchronous = false