You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Guillermo Muñoz Salgado <mu...@gmail.com> on 2022/02/03 15:57:38 UTC
ListSFTP doesn't follow symlinks
Hi all,
We are developing a use case in which we have to get some files from a
server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes
cluster running nifi 1.15.3. But we are having some issues when what
we want to get are symlinks instead of files. We have set true the
property *"Follow
symlink" *but we get the same results. Are we doing something wrong? Or is
it a bug or a known issue? We have found this issue [1] but it is old and
resolved and this other one [2], that is older and unresolved. We're not
sure if they are related to this behaviour or not.
I paste our error log:
2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18]
o.a.nifi.processors.standard.GetSFTP
GetSFTP[id=c0300c77-017e-1000-ffff-fff-ffa9c1f31] Unable to get listing
from testfile.gz; skipping: java.io.FileNotFoundException: Could not
perform listing on testfile.gz because could not find the file on the
remote server
java.io.FileNotFoundException: Could not perform listing on testfile.gz
because could not find the file on the remote server
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
at
org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
at
org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at
org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thanks in advance
--
Guille
[1] https://issues.apache.org/jira/browse/NIFI-5560
[2] https://issues.apache.org/jira/browse/NIFI-6699
Re: ListSFTP doesn't follow symlinks
Posted by Guillermo Muñoz Salgado <mu...@gmail.com>.
Thanks Mark and David.
I will watch the Issue at Jira to be updated.
Thanks again!
Regards.
El jue, 3 feb 2022 a las 18:28, David Handermann (<
exceptionfactory@apache.org>) escribió:
> Guille,
>
> Thanks for confirming the same behavior on ListSFTP, and thanks Mark for
> reproducing the issue.
>
> Apparently the problem is in the NiFi SFTPTransfer class, which is not
> distinguishing between a symbolic link for a file and for a directory. In
> this case, SFTPTransfer is attempting to process the symlinked file as a
> directory, causing the error. I have assigned NIFI-6699 and will plan on
> submitting a pull request to resolve the problem soon.
>
> Thanks again for reporting this issue and helping track down the problem!
>
> Regards,
> David Handermann
>
> On Thu, Feb 3, 2022 at 10:58 AM Mark Payne <ma...@hotmail.com> wrote:
>
>> Guille,
>>
>> Thanks for the extra details.
>>
>> I just tried again. In my case, all worked as expected when I had a
>> symlink to a directory. But when I had a symlink to a file, I got the same
>> error and stack trace as you. So looks like we are handling the case
>> properly for symlinked directories but not symlinked files.
>>
>> Thanks
>> -Mark
>>
>>
>>
>> On Feb 3, 2022, at 11:43 AM, Guillermo Muñoz <
>> guillermo.munoz.salgado@gmail.com> wrote:
>>
>> Hi, David.
>>
>> Sorry for the misunderstanding, my fault. Firstly, we tried using
>> ListSFTP and FetchSFTP, and when it didn't work, we tried another option
>> (GetSFTP), and I pasted the wrong stack trace. So, i've done the following
>> tests:
>>
>> - ListSFTP + FetchSFTP: Error in ListSFTP [1]
>> - GetSFTP: Error [2]
>> - Generate flowfile + FetchSFTP with the name of the symlink in the *
>> Remote File* property: OK, the file is downloaded.
>>
>> So, it seems the issue is in ListSFTP and GetSFTP, but FetchSFTP works
>> fine.
>>
>> Thanks. Regards
>>
>> --
>> Guille
>>
>> [1]
>> 2022-02-03 17:36:45,466 ERROR [Timer-Driven Process Thread-8]
>> o.a.nifi.processors.standard.ListSFTP
>> ListSFTP[id=64443154-ac76-1736-9e49-f2ca388dfbdf] Unable to get listing
>> from *****.gz; skipping: java.io.FileNotFoundException: Could not
>> perform listing on *****.gz because could not find the file on the
>> remote server
>> java.io.FileNotFoundException: Could not perform listing on *****.gz
>> because could not find the file on the remote server
>> at
>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
>> at
>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
>> at
>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
>> at
>> org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120)
>> at
>> org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:150)
>> at
>> org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112)
>> at
>> org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:750)
>> at
>> org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525)
>> at
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
>> at
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
>> at
>> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> [2]
>> 2022-02-03 17:39:14,714 ERROR [Timer-Driven Process Thread-27]
>> o.a.nifi.processors.standard.GetSFTP
>> GetSFTP[id=c0300c77-017e-1000-ffff-fffffa9c1f31] Unable to get listing from
>> *****.gz; skipping: java.io.FileNotFoundException: Could not perform
>> listing on *****.gz because could not find the file on the remote server
>> java.io.FileNotFoundException: Could not perform listing on *****.gz
>> because could not find the file on the remote server
>> at
>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
>> at
>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
>> at
>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
>> at
>> org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
>> at
>> org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
>> at
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
>> at
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
>> at
>> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> El jue, 3 feb 2022 a las 17:09, David Handermann (<
>> exceptionfactory@apache.org>) escribió:
>>
>>> Hi Guille,
>>>
>>> Thanks for raising this issue and providing a stack trace. You
>>> mentioned using ListSFTP and FetchSFTP, but the stack references
>>> GetFileTransfer, which corresponds to GetSFTP.
>>>
>>> Can you confirm the same error using FetchSFTP? If you can confirm the
>>> same issue with FetchSFTP, it would be very helpful to add those details on
>>> the newer Jira issue NIFI-6699.
>>>
>>> NiFi SFTP processors switched to a different SSH library after the
>>> resolution of NIFI-5560, so it is possible that some changes may be
>>> necessary. However, it would be helpful to confirm whether this is an
>>> issue with FetchSFTP, GetSFTP, or both processors.
>>>
>>> Regards,
>>> David Handermann
>>>
>>> On Thu, Feb 3, 2022 at 9:57 AM Guillermo Muñoz Salgado <mu...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> We are developing a use case in which we have to get some files from a
>>>> server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes
>>>> cluster running nifi 1.15.3. But we are having some issues when what
>>>> we want to get are symlinks instead of files. We have set true the property *"Follow
>>>> symlink" *but we get the same results. Are we doing something wrong?
>>>> Or is it a bug or a known issue? We have found this issue [1] but it is old
>>>> and resolved and this other one [2], that is older and unresolved. We're
>>>> not sure if they are related to this behaviour or not.
>>>>
>>>> I paste our error log:
>>>>
>>>> 2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18]
>>>> o.a.nifi.processors.standard.GetSFTP
>>>> GetSFTP[id=c0300c77-017e-1000-ffff-fff-ffa9c1f31] Unable to get listing
>>>> from testfile.gz; skipping: java.io.FileNotFoundException: Could not
>>>> perform listing on testfile.gz because could not find the file on the
>>>> remote server
>>>> java.io.FileNotFoundException: Could not perform listing on testfile.gz
>>>> because could not find the file on the remote server
>>>> at
>>>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
>>>> at
>>>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
>>>> at
>>>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
>>>> at
>>>> org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
>>>> at
>>>> org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
>>>> at
>>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>>> at
>>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
>>>> at
>>>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
>>>> at
>>>> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
>>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>> at
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>>
>>>> Thanks in advance
>>>> --
>>>> Guille
>>>>
>>>> [1] https://issues.apache.org/jira/browse/NIFI-5560
>>>> [2] https://issues.apache.org/jira/browse/NIFI-6699
>>>>
>>>
>>
>> --
>> Guille
>>
>>
>>
--
Guille
Re: ListSFTP doesn't follow symlinks
Posted by David Handermann <ex...@apache.org>.
Guille,
Thanks for confirming the same behavior on ListSFTP, and thanks Mark for
reproducing the issue.
Apparently the problem is in the NiFi SFTPTransfer class, which is not
distinguishing between a symbolic link for a file and for a directory. In
this case, SFTPTransfer is attempting to process the symlinked file as a
directory, causing the error. I have assigned NIFI-6699 and will plan on
submitting a pull request to resolve the problem soon.
Thanks again for reporting this issue and helping track down the problem!
Regards,
David Handermann
On Thu, Feb 3, 2022 at 10:58 AM Mark Payne <ma...@hotmail.com> wrote:
> Guille,
>
> Thanks for the extra details.
>
> I just tried again. In my case, all worked as expected when I had a
> symlink to a directory. But when I had a symlink to a file, I got the same
> error and stack trace as you. So looks like we are handling the case
> properly for symlinked directories but not symlinked files.
>
> Thanks
> -Mark
>
>
>
> On Feb 3, 2022, at 11:43 AM, Guillermo Muñoz <
> guillermo.munoz.salgado@gmail.com> wrote:
>
> Hi, David.
>
> Sorry for the misunderstanding, my fault. Firstly, we tried using ListSFTP
> and FetchSFTP, and when it didn't work, we tried another option (GetSFTP),
> and I pasted the wrong stack trace. So, i've done the following tests:
>
> - ListSFTP + FetchSFTP: Error in ListSFTP [1]
> - GetSFTP: Error [2]
> - Generate flowfile + FetchSFTP with the name of the symlink in the *
> Remote File* property: OK, the file is downloaded.
>
> So, it seems the issue is in ListSFTP and GetSFTP, but FetchSFTP works
> fine.
>
> Thanks. Regards
>
> --
> Guille
>
> [1]
> 2022-02-03 17:36:45,466 ERROR [Timer-Driven Process Thread-8]
> o.a.nifi.processors.standard.ListSFTP
> ListSFTP[id=64443154-ac76-1736-9e49-f2ca388dfbdf] Unable to get listing
> from *****.gz; skipping: java.io.FileNotFoundException: Could not
> perform listing on *****.gz because could not find the file on the
> remote server
> java.io.FileNotFoundException: Could not perform listing on *****.gz
> because could not find the file on the remote server
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
> at
> org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120)
> at
> org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:150)
> at
> org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112)
> at
> org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:750)
> at
> org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
> at
> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
> [2]
> 2022-02-03 17:39:14,714 ERROR [Timer-Driven Process Thread-27]
> o.a.nifi.processors.standard.GetSFTP
> GetSFTP[id=c0300c77-017e-1000-ffff-fffffa9c1f31] Unable to get listing from
> *****.gz; skipping: java.io.FileNotFoundException: Could not perform
> listing on *****.gz because could not find the file on the remote server
> java.io.FileNotFoundException: Could not perform listing on *****.gz
> because could not find the file on the remote server
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
> at
> org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
> at
> org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
> at
> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> El jue, 3 feb 2022 a las 17:09, David Handermann (<
> exceptionfactory@apache.org>) escribió:
>
>> Hi Guille,
>>
>> Thanks for raising this issue and providing a stack trace. You mentioned
>> using ListSFTP and FetchSFTP, but the stack references GetFileTransfer,
>> which corresponds to GetSFTP.
>>
>> Can you confirm the same error using FetchSFTP? If you can confirm the
>> same issue with FetchSFTP, it would be very helpful to add those details on
>> the newer Jira issue NIFI-6699.
>>
>> NiFi SFTP processors switched to a different SSH library after the
>> resolution of NIFI-5560, so it is possible that some changes may be
>> necessary. However, it would be helpful to confirm whether this is an
>> issue with FetchSFTP, GetSFTP, or both processors.
>>
>> Regards,
>> David Handermann
>>
>> On Thu, Feb 3, 2022 at 9:57 AM Guillermo Muñoz Salgado <mu...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> We are developing a use case in which we have to get some files from a
>>> server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes
>>> cluster running nifi 1.15.3. But we are having some issues when what
>>> we want to get are symlinks instead of files. We have set true the property *"Follow
>>> symlink" *but we get the same results. Are we doing something wrong? Or
>>> is it a bug or a known issue? We have found this issue [1] but it is old
>>> and resolved and this other one [2], that is older and unresolved. We're
>>> not sure if they are related to this behaviour or not.
>>>
>>> I paste our error log:
>>>
>>> 2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18]
>>> o.a.nifi.processors.standard.GetSFTP
>>> GetSFTP[id=c0300c77-017e-1000-ffff-fff-ffa9c1f31] Unable to get listing
>>> from testfile.gz; skipping: java.io.FileNotFoundException: Could not
>>> perform listing on testfile.gz because could not find the file on the
>>> remote server
>>> java.io.FileNotFoundException: Could not perform listing on testfile.gz
>>> because could not find the file on the remote server
>>> at
>>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
>>> at
>>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
>>> at
>>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
>>> at
>>> org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
>>> at
>>> org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
>>> at
>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> at
>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
>>> at
>>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
>>> at
>>> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>> Thanks in advance
>>> --
>>> Guille
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-5560
>>> [2] https://issues.apache.org/jira/browse/NIFI-6699
>>>
>>
>
> --
> Guillermo Muñoz Salgado
>
>
>
Re: ListSFTP doesn't follow symlinks
Posted by Mark Payne <ma...@hotmail.com>.
Guille,
Thanks for the extra details.
I just tried again. In my case, all worked as expected when I had a symlink to a directory. But when I had a symlink to a file, I got the same error and stack trace as you. So looks like we are handling the case properly for symlinked directories but not symlinked files.
Thanks
-Mark
On Feb 3, 2022, at 11:43 AM, Guillermo Muñoz <gu...@gmail.com>> wrote:
Hi, David.
Sorry for the misunderstanding, my fault. Firstly, we tried using ListSFTP and FetchSFTP, and when it didn't work, we tried another option (GetSFTP), and I pasted the wrong stack trace. So, i've done the following tests:
* ListSFTP + FetchSFTP: Error in ListSFTP [1]
* GetSFTP: Error [2]
* Generate flowfile + FetchSFTP with the name of the symlink in the Remote File property: OK, the file is downloaded.
So, it seems the issue is in ListSFTP and GetSFTP, but FetchSFTP works fine.
Thanks. Regards
--
Guille
[1]
2022-02-03 17:36:45,466 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=64443154-ac76-1736-9e49-f2ca388dfbdf] Unable to get listing from *****.gz; skipping: java.io.FileNotFoundException: Could not perform listing on *****.gz because could not find the file on the remote server
java.io.FileNotFoundException: Could not perform listing on *****.gz because could not find the file on the remote server
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
at org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120)
at org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:150)
at org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112)
at org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:750)
at org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2]
2022-02-03 17:39:14,714 ERROR [Timer-Driven Process Thread-27] o.a.nifi.processors.standard.GetSFTP GetSFTP[id=c0300c77-017e-1000-ffff-fffffa9c1f31] Unable to get listing from *****.gz; skipping: java.io.FileNotFoundException: Could not perform listing on *****.gz because could not find the file on the remote server
java.io.FileNotFoundException: Could not perform listing on *****.gz because could not find the file on the remote server
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
at org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
at org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
El jue, 3 feb 2022 a las 17:09, David Handermann (<ex...@apache.org>>) escribió:
Hi Guille,
Thanks for raising this issue and providing a stack trace. You mentioned using ListSFTP and FetchSFTP, but the stack references GetFileTransfer, which corresponds to GetSFTP.
Can you confirm the same error using FetchSFTP? If you can confirm the same issue with FetchSFTP, it would be very helpful to add those details on the newer Jira issue NIFI-6699.
NiFi SFTP processors switched to a different SSH library after the resolution of NIFI-5560, so it is possible that some changes may be necessary. However, it would be helpful to confirm whether this is an issue with FetchSFTP, GetSFTP, or both processors.
Regards,
David Handermann
On Thu, Feb 3, 2022 at 9:57 AM Guillermo Muñoz Salgado <mu...@gmail.com>> wrote:
Hi all,
We are developing a use case in which we have to get some files from a server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes cluster running nifi 1.15.3. But we are having some issues when what we want to get are symlinks instead of files. We have set true the property "Follow symlink" but we get the same results. Are we doing something wrong? Or is it a bug or a known issue? We have found this issue [1] but it is old and resolved and this other one [2], that is older and unresolved. We're not sure if they are related to this behaviour or not.
I paste our error log:
2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18] o.a.nifi.processors.standard.GetSFTP GetSFTP[id=c0300c77-017e-1000-ffff-fff-ffa9c1f31] Unable to get listing from testfile.gz; skipping: java.io.FileNotFoundException: Could not perform listing on testfile.gz because could not find the file on the remote server
java.io.FileNotFoundException: Could not perform listing on testfile.gz because could not find the file on the remote server
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
at org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
at org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thanks in advance
--
Guille
[1] https://issues.apache.org/jira/browse/NIFI-5560
[2] https://issues.apache.org/jira/browse/NIFI-6699
--
Guillermo Muñoz Salgado
Re: ListSFTP doesn't follow symlinks
Posted by Guillermo Muñoz <gu...@gmail.com>.
Hi, David.
Sorry for the misunderstanding, my fault. Firstly, we tried using ListSFTP
and FetchSFTP, and when it didn't work, we tried another option (GetSFTP),
and I pasted the wrong stack trace. So, i've done the following tests:
- ListSFTP + FetchSFTP: Error in ListSFTP [1]
- GetSFTP: Error [2]
- Generate flowfile + FetchSFTP with the name of the symlink in the *Remote
File* property: OK, the file is downloaded.
So, it seems the issue is in ListSFTP and GetSFTP, but FetchSFTP works
fine.
Thanks. Regards
--
Guille
[1]
2022-02-03 17:36:45,466 ERROR [Timer-Driven Process Thread-8]
o.a.nifi.processors.standard.ListSFTP
ListSFTP[id=64443154-ac76-1736-9e49-f2ca388dfbdf] Unable to get listing
from *****.gz; skipping: java.io.FileNotFoundException: Could not perform
listing on *****.gz because could not find the file on the remote server
java.io.FileNotFoundException: Could not perform listing on *****.gz
because could not find the file on the remote server
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
at
org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120)
at
org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:150)
at
org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112)
at
org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:750)
at
org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at
org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2]
2022-02-03 17:39:14,714 ERROR [Timer-Driven Process Thread-27]
o.a.nifi.processors.standard.GetSFTP
GetSFTP[id=c0300c77-017e-1000-ffff-fffffa9c1f31] Unable to get listing from
*****.gz; skipping: java.io.FileNotFoundException: Could not perform
listing on *****.gz because could not find the file on the remote server
java.io.FileNotFoundException: Could not perform listing on *****.gz
because could not find the file on the remote server
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
at
org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
at
org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
at
org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at
org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
El jue, 3 feb 2022 a las 17:09, David Handermann (<
exceptionfactory@apache.org>) escribió:
> Hi Guille,
>
> Thanks for raising this issue and providing a stack trace. You mentioned
> using ListSFTP and FetchSFTP, but the stack references GetFileTransfer,
> which corresponds to GetSFTP.
>
> Can you confirm the same error using FetchSFTP? If you can confirm the
> same issue with FetchSFTP, it would be very helpful to add those details on
> the newer Jira issue NIFI-6699.
>
> NiFi SFTP processors switched to a different SSH library after the
> resolution of NIFI-5560, so it is possible that some changes may be
> necessary. However, it would be helpful to confirm whether this is an
> issue with FetchSFTP, GetSFTP, or both processors.
>
> Regards,
> David Handermann
>
> On Thu, Feb 3, 2022 at 9:57 AM Guillermo Muñoz Salgado <mu...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> We are developing a use case in which we have to get some files from a
>> server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes
>> cluster running nifi 1.15.3. But we are having some issues when what
>> we want to get are symlinks instead of files. We have set true the property *"Follow
>> symlink" *but we get the same results. Are we doing something wrong? Or
>> is it a bug or a known issue? We have found this issue [1] but it is old
>> and resolved and this other one [2], that is older and unresolved. We're
>> not sure if they are related to this behaviour or not.
>>
>> I paste our error log:
>>
>> 2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18]
>> o.a.nifi.processors.standard.GetSFTP
>> GetSFTP[id=c0300c77-017e-1000-ffff-fff-ffa9c1f31] Unable to get listing
>> from testfile.gz; skipping: java.io.FileNotFoundException: Could not
>> perform listing on testfile.gz because could not find the file on the
>> remote server
>> java.io.FileNotFoundException: Could not perform listing on testfile.gz
>> because could not find the file on the remote server
>> at
>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
>> at
>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
>> at
>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
>> at
>> org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
>> at
>> org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
>> at
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
>> at
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
>> at
>> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>> Thanks in advance
>> --
>> Guille
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-5560
>> [2] https://issues.apache.org/jira/browse/NIFI-6699
>>
>
--
Guillermo Muñoz Salgado
Re: ListSFTP doesn't follow symlinks
Posted by David Handermann <ex...@apache.org>.
Hi Guille,
Thanks for raising this issue and providing a stack trace. You mentioned
using ListSFTP and FetchSFTP, but the stack references GetFileTransfer,
which corresponds to GetSFTP.
Can you confirm the same error using FetchSFTP? If you can confirm the
same issue with FetchSFTP, it would be very helpful to add those details on
the newer Jira issue NIFI-6699.
NiFi SFTP processors switched to a different SSH library after the
resolution of NIFI-5560, so it is possible that some changes may be
necessary. However, it would be helpful to confirm whether this is an
issue with FetchSFTP, GetSFTP, or both processors.
Regards,
David Handermann
On Thu, Feb 3, 2022 at 9:57 AM Guillermo Muñoz Salgado <mu...@gmail.com>
wrote:
> Hi all,
>
> We are developing a use case in which we have to get some files from a
> server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes
> cluster running nifi 1.15.3. But we are having some issues when what
> we want to get are symlinks instead of files. We have set true the property *"Follow
> symlink" *but we get the same results. Are we doing something wrong? Or
> is it a bug or a known issue? We have found this issue [1] but it is old
> and resolved and this other one [2], that is older and unresolved. We're
> not sure if they are related to this behaviour or not.
>
> I paste our error log:
>
> 2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18]
> o.a.nifi.processors.standard.GetSFTP
> GetSFTP[id=c0300c77-017e-1000-ffff-fff-ffa9c1f31] Unable to get listing
> from testfile.gz; skipping: java.io.FileNotFoundException: Could not
> perform listing on testfile.gz because could not find the file on the
> remote server
> java.io.FileNotFoundException: Could not perform listing on testfile.gz
> because could not find the file on the remote server
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
> at
> org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
> at
> org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
> at
> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> Thanks in advance
> --
> Guille
>
> [1] https://issues.apache.org/jira/browse/NIFI-5560
> [2] https://issues.apache.org/jira/browse/NIFI-6699
>
Re: ListSFTP doesn't follow symlinks
Posted by Mark Payne <ma...@hotmail.com>.
Guille,
I did a quick test on my MacBook and things worked as expected following symlinks. If I set the property to “false” it didn’t get the files. If I set it to “true” it did retrieve the files. Either way it didn’t error, though - just didn’t follow the symlink.
Of course, that’s not to say that there’s not some issue, just that it’s not obviously always broken :)
One thing that I notice in the log message there, though:
"Could not perform listing on testfile.gz…”
There are a couple of spaces there - it’s not looking for “testfile.gz” but rather “<space><space>testfile.gz” - are there actually spaces in the filename? Or any other sort of character there, that is perhaps not being properly escaped?
Thanks
-Mark
On Feb 3, 2022, at 10:57 AM, Guillermo Muñoz Salgado <mu...@gmail.com>> wrote:
Hi all,
We are developing a use case in which we have to get some files from a server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes cluster running nifi 1.15.3. But we are having some issues when what we want to get are symlinks instead of files. We have set true the property "Follow symlink" but we get the same results. Are we doing something wrong? Or is it a bug or a known issue? We have found this issue [1] but it is old and resolved and this other one [2], that is older and unresolved. We're not sure if they are related to this behaviour or not.
I paste our error log:
2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18] o.a.nifi.processors.standard.GetSFTP GetSFTP[id=c0300c77-017e-1000-ffff-fff-ffa9c1f31] Unable to get listing from testfile.gz; skipping: java.io.FileNotFoundException: Could not perform listing on testfile.gz because could not find the file on the remote server
java.io.FileNotFoundException: Could not perform listing on testfile.gz because could not find the file on the remote server
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262)
at org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299)
at org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Thanks in advance
--
Guille
[1] https://issues.apache.org/jira/browse/NIFI-5560
[2] https://issues.apache.org/jira/browse/NIFI-6699