You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Vibhath Ileperuma <vi...@gmail.com> on 2021/08/17 05:52:49 UTC
FetchS3Object fails to fetch small files from S3
Hi All,
I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The
'FetchS3Object' processor throws the following error only for small files.
I tried to fetch the same set of files using Nifi 1.13.2. There was no
issue with 1.13.2. Can you please let me know a way to overcome this issue.
FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to retrieve
S3 Object for
StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1629123317469-1, container=default,
section=1], offset=10164,
length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1];
routing to failure: The requested range is not satisfiable (Service: Amazon
S3; Status Code: 416; Error Code: InvalidRange; Request ID:
NDCXP1G52A96HJ2R; S3 Extended Request ID:
KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=;
Proxy: null)
Thank You.
Vibhath
Re: FetchS3Object fails to fetch small files from S3
Posted by Joey Frazee <jo...@icloud.com>.
I was able to replicate this on Minio too and have a fix for it. The change is trivial but it needs more testing.
-joey
> On Aug 17, 2021, at 8:22 AM, Vibhath Ileperuma <vi...@gmail.com> wrote:
>
>
> Hi Joey,
>
> It happens only for 0B files.
> Sorry for the confusion caused.
>
> Vibhath.
Re: FetchS3Object fails to fetch small files from S3
Posted by Vibhath Ileperuma <vi...@gmail.com>.
Hi Joey,
It happens only for 0B files.
Sorry for the confusion caused.
Vibhath.
Re: FetchS3Object fails to fetch small files from S3
Posted by Joey Frazee <jo...@icloud.com>.
Arpad, the validator on range length should prevent that from happening if there’s no EL. Not set or >= 1 should be the only allowed values.
Vibhath, does this happen with small, non-zero length files?
FYI, opened https://issues.apache.org/jira/browse/NIFI-9055
-joey
> On Aug 17, 2021, at 4:41 AM, Arpad Boda <ab...@apache.org> wrote:
>
>
> final Long rangeLength = (context.getProperty(RANGE_LENGTH).isSet() ? context.getProperty(RANGE_LENGTH).evaluateAttributeExpressions(flowFile).asDataSize(DataUnit.B).longValue() : null);
>
> if (rangeLength != null) {
> request.setRange(rangeStart, rangeStart + rangeLength - 1);
> }
>
> I think the problem is that rangeLength can be 0 when it's not null, so the request becomes invalid.
>
>
>
>> On Tue, Aug 17, 2021 at 1:19 PM Vibhath Ileperuma <vi...@gmail.com> wrote:
>> Hi Joey and Denes,
>>
>> I'm also getting this error for 0 byte files.
>>
>> Thank You.
>>
>>> On Tue, Aug 17, 2021 at 12:25 PM Denes Arvay <de...@apache.org> wrote:
>>> Joey,
>>>
>>> I was able to reproduce the issue with a 0 byte file.
>>> Reverting the commit you were referring to ([1]) fixes the problem.
>>>
>>> Denes
>>>
>>> [1] https://github.com/apache/nifi/commit/0ed35345245da057400a7d875e47f57bc2d3794d
>>>
>>>> On Tue, Aug 17, 2021 at 8:12 AM Joey Frazee <jo...@icloud.com> wrote:
>>>> Vibhath, there was a change to allow reading by ranges that I’m wondering if is causing this.
>>>>
>>>> When you say small, exactly how small are the files?
>>>>
>>>> -joey
>>>>
>>>>>> On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma <vi...@gmail.com> wrote:
>>>>>>
>>>>>
>>>>> Hi All,
>>>>>
>>>>> I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The 'FetchS3Object' processor throws the following error only for small files. I tried to fetch the same set of files using Nifi 1.13.2. There was no issue with 1.13.2. Can you please let me know a way to overcome this issue.
>>>>>
>>>>> FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to retrieve S3 Object for StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1629123317469-1, container=default, section=1], offset=10164, length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1]; routing to failure: The requested range is not satisfiable (Service: Amazon S3; Status Code: 416; Error Code: InvalidRange; Request ID: NDCXP1G52A96HJ2R; S3 Extended Request ID: KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=; Proxy: null)
>>>>>
>>>>> Thank You.
>>>>> Vibhath
>>>>>
Re: FetchS3Object fails to fetch small files from S3
Posted by Arpad Boda <ab...@apache.org>.
final Long rangeLength = (context.getProperty(RANGE_LENGTH).isSet() ?
context.getProperty(RANGE_LENGTH).evaluateAttributeExpressions(flowFile).asDataSize(DataUnit.B).longValue()
: null);
if (rangeLength != null) {
request.setRange(rangeStart, rangeStart + rangeLength - 1);
}
I think the problem is that rangeLength can be 0 when it's not null, so the
request becomes invalid.
On Tue, Aug 17, 2021 at 1:19 PM Vibhath Ileperuma <
vibhatharunapriya@gmail.com> wrote:
> Hi Joey and Denes,
>
> I'm also getting this error for 0 byte files.
>
> Thank You.
>
> On Tue, Aug 17, 2021 at 12:25 PM Denes Arvay <de...@apache.org> wrote:
>
>> Joey,
>>
>> I was able to reproduce the issue with a 0 byte file.
>> Reverting the commit you were referring to ([1]) fixes the problem.
>>
>> Denes
>>
>> [1]
>> https://github.com/apache/nifi/commit/0ed35345245da057400a7d875e47f57bc2d3794d
>>
>> On Tue, Aug 17, 2021 at 8:12 AM Joey Frazee <jo...@icloud.com>
>> wrote:
>>
>>> Vibhath, there was a change to allow reading by ranges that I’m
>>> wondering if is causing this.
>>>
>>> When you say small, exactly how small are the files?
>>>
>>> -joey
>>>
>>> On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma <
>>> vibhatharunapriya@gmail.com> wrote:
>>>
>>>
>>> Hi All,
>>>
>>> I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The
>>> 'FetchS3Object' processor throws the following error only for small files.
>>> I tried to fetch the same set of files using Nifi 1.13.2. There was no
>>> issue with 1.13.2. Can you please let me know a way to overcome this issue.
>>>
>>> FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to
>>> retrieve S3 Object for
>>> StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim
>>> [resourceClaim=StandardResourceClaim[id=1629123317469-1, container=default,
>>> section=1], offset=10164,
>>> length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1];
>>> routing to failure: The requested range is not satisfiable (Service: Amazon
>>> S3; Status Code: 416; Error Code: InvalidRange; Request ID:
>>> NDCXP1G52A96HJ2R; S3 Extended Request ID:
>>> KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=;
>>> Proxy: null)
>>>
>>> Thank You.
>>> Vibhath
>>>
>>>
>>>
Re: FetchS3Object fails to fetch small files from S3
Posted by Vibhath Ileperuma <vi...@gmail.com>.
Hi Joey and Denes,
I'm also getting this error for 0 byte files.
Thank You.
On Tue, Aug 17, 2021 at 12:25 PM Denes Arvay <de...@apache.org> wrote:
> Joey,
>
> I was able to reproduce the issue with a 0 byte file.
> Reverting the commit you were referring to ([1]) fixes the problem.
>
> Denes
>
> [1]
> https://github.com/apache/nifi/commit/0ed35345245da057400a7d875e47f57bc2d3794d
>
> On Tue, Aug 17, 2021 at 8:12 AM Joey Frazee <jo...@icloud.com>
> wrote:
>
>> Vibhath, there was a change to allow reading by ranges that I’m wondering
>> if is causing this.
>>
>> When you say small, exactly how small are the files?
>>
>> -joey
>>
>> On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma <
>> vibhatharunapriya@gmail.com> wrote:
>>
>>
>> Hi All,
>>
>> I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The
>> 'FetchS3Object' processor throws the following error only for small files.
>> I tried to fetch the same set of files using Nifi 1.13.2. There was no
>> issue with 1.13.2. Can you please let me know a way to overcome this issue.
>>
>> FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to
>> retrieve S3 Object for
>> StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1629123317469-1, container=default,
>> section=1], offset=10164,
>> length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1];
>> routing to failure: The requested range is not satisfiable (Service: Amazon
>> S3; Status Code: 416; Error Code: InvalidRange; Request ID:
>> NDCXP1G52A96HJ2R; S3 Extended Request ID:
>> KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=;
>> Proxy: null)
>>
>> Thank You.
>> Vibhath
>>
>>
>>
Re: FetchS3Object fails to fetch small files from S3
Posted by Denes Arvay <de...@apache.org>.
Joey,
I was able to reproduce the issue with a 0 byte file.
Reverting the commit you were referring to ([1]) fixes the problem.
Denes
[1]
https://github.com/apache/nifi/commit/0ed35345245da057400a7d875e47f57bc2d3794d
On Tue, Aug 17, 2021 at 8:12 AM Joey Frazee <jo...@icloud.com> wrote:
> Vibhath, there was a change to allow reading by ranges that I’m wondering
> if is causing this.
>
> When you say small, exactly how small are the files?
>
> -joey
>
> On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma <
> vibhatharunapriya@gmail.com> wrote:
>
>
> Hi All,
>
> I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The
> 'FetchS3Object' processor throws the following error only for small files.
> I tried to fetch the same set of files using Nifi 1.13.2. There was no
> issue with 1.13.2. Can you please let me know a way to overcome this issue.
>
> FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to retrieve
> S3 Object for
> StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1629123317469-1, container=default,
> section=1], offset=10164,
> length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1];
> routing to failure: The requested range is not satisfiable (Service: Amazon
> S3; Status Code: 416; Error Code: InvalidRange; Request ID:
> NDCXP1G52A96HJ2R; S3 Extended Request ID:
> KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=;
> Proxy: null)
>
> Thank You.
> Vibhath
>
>
>
Re: FetchS3Object fails to fetch small files from S3
Posted by Joey Frazee <jo...@icloud.com>.
Vibhath, there was a change to allow reading by ranges that I’m wondering if is causing this.
When you say small, exactly how small are the files?
-joey
> On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma <vi...@gmail.com> wrote:
>
>
> Hi All,
>
> I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The 'FetchS3Object' processor throws the following error only for small files. I tried to fetch the same set of files using Nifi 1.13.2. There was no issue with 1.13.2. Can you please let me know a way to overcome this issue.
>
> FetchS3Object[Id: 684e3935-ddb1-1004-dd60-2e926d9469a4] Failed to retrieve S3 Object for StandardFlowFileRecord[uuid=6e880507-4f49-40fe-a447-de66cb4bc1f3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1629123317469-1, container=default, section=1], offset=10164, length=2],offset=1,name=54fa8086-32df-4257-93cf-ca18a5ecc615,size=1]; routing to failure: The requested range is not satisfiable (Service: Amazon S3; Status Code: 416; Error Code: InvalidRange; Request ID: NDCXP1G52A96HJ2R; S3 Extended Request ID: KbUDb+zoH78kP9/lqDo4DMj8P53kHjdhdPF4EnsfMcwtBs3Qse/UK4otNvyDrgtiECXZ9f8YaJk=; Proxy: null)
>
> Thank You.
> Vibhath
>