You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Aeden Jameson <ae...@gmail.com> on 2022/05/11 23:53:41 UTC
Checkpointing - Job Manager S3 Head Requests are 404s
We're using S3 to store checkpoints. They are taken every minute. I'm
seeing a large number of 404 responses from S3 being generated by the
job manager. The order of the entries in the debugging log would imply
that it's a result of a HEAD request to a key. For example all the
incidents look like this,
2022-05-11 23:29:00,804 DEBUG com.amazonaws.request [] - Sending
Request: HEAD https://[MY-BUCKET].s3.amazonaws.com
/[MY_JOB]/checkpoints/5f4d6923883a1702b206f978fa3637a3/ Headers:
(amz-sdk-invocation-id: XXXXX, Content-Type: application/octet-stream,
User-Agent: Hadoop 3.1.0, aws-sdk-java/1.11.788
Linux/5.4.181-99.354.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.13+8
java/11.0.13 scala/2.12.7 vendor/Oracle_Corporation, )
2022-05-11 23:29:00,815 DEBUG com.amazonaws.request [] - Received
error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not
Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not
Found; ......)
The key does in fact exist. How can I go about resolving this?
--
Cheers,
Aeden
GitHub: https://github.com/aedenj
Re: Checkpointing - Job Manager S3 Head Requests are 404s
Posted by David Anderson <da...@apache.org>.
One more thing to be aware of: the Presto S3 implementation has issues too.
See FLINK-24392 [1]. This means that there's no ideal solution, and in some
cases it is preferable to use Hadoop, perhaps in combination with
increasing the value of state.storage.fs.memory-threshold [2] in order to
decrease the number of data files written with each checkpoint.
[1] https://issues.apache.org/jira/browse/FLINK-24392
[2]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/#state-storage-fs-memory-threshold
On Thu, May 19, 2022 at 8:45 PM Aeden Jameson <ae...@gmail.com>
wrote:
> Great, that all makes sense to me. Thanks again.
>
> On Thu, May 19, 2022 at 11:42 AM David Anderson <da...@apache.org>
> wrote:
> >
> > Sure, happy to try to help.
> >
> > What's happening with the hadoop filesystem is that before it writes
> each key it checks to see if the "parent directory" exists by checking for
> a key with the prefix up to the last "/", and if that key isn't found it
> then creates empty marker files to cause of that parent directory to exist.
> These existence checks are S3 HEAD requests. None of this is helpful in the
> case of Flink checkpointing.
> >
> > And yes, while Presto avoids this unnecessary overhead, entropy
> injection remains an important tool for improving scalability. Where you'll
> run into quotas and rate limits depends, of course, on factors like the
> number of stateful tasks in your job(s), the parallelism, and the
> checkpointing interval.
> >
> > On Thu, May 19, 2022 at 8:04 PM Aeden Jameson <ae...@gmail.com>
> wrote:
> >>
> >> Thanks for the response David. That's the conclusion I came to as
> >> well. The Hadoop plugin behavior doesn't appear to reflect more
> >> recent changes to S3 like strong read-after-write consistency,
> >>
> https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/
> >> .
> >>
> >> Given the improved interaction of the Presto plugin with S3 I would
> >> conclude that this increases the size of the cluster that would
> >> require entropy injection, yes? But that it doesn't necessarily get
> >> rid of the need because one could have a large enough cluster and say
> >> a lifecycle policy that could still end up requiring entropy
> >> injection.
> >>
> >> On Thu, May 19, 2022 at 10:24 AM David Anderson <da...@apache.org>
> wrote:
> >> >
> >> > Aeden, this is probably happening because you are using the Hadoop
> implementation of S3.
> >> >
> >> > The Hadoop S3 filesystem tries to imitate a filesystem on top of S3.
> In so doing it makes a lot of HEAD requests. These are expensive, and they
> violate read-after-create visibility, which is what you seem to be
> experiencing. By contrast, the Presto S3 implementation doesn't do the same
> (harmful in this case) magic, and simply does PUT/GET operations. Because
> that's all Flink needs to checkpointing, this works much better.
> >> >
> >> > Best,
> >> > David
> >> >
> >> > On Thu, May 12, 2022 at 1:53 AM Aeden Jameson <
> aeden.jameson@gmail.com> wrote:
> >> >>
> >> >> We're using S3 to store checkpoints. They are taken every minute. I'm
> >> >> seeing a large number of 404 responses from S3 being generated by the
> >> >> job manager. The order of the entries in the debugging log would
> imply
> >> >> that it's a result of a HEAD request to a key. For example all the
> >> >> incidents look like this,
> >> >>
> >> >>
> >> >> 2022-05-11 23:29:00,804 DEBUG com.amazonaws.request [] - Sending
> >> >> Request: HEAD https://[MY-BUCKET].s3.amazonaws.com
> >> >> /[MY_JOB]/checkpoints/5f4d6923883a1702b206f978fa3637a3/ Headers:
> >> >> (amz-sdk-invocation-id: XXXXX, Content-Type:
> application/octet-stream,
> >> >> User-Agent: Hadoop 3.1.0, aws-sdk-java/1.11.788
> >> >> Linux/5.4.181-99.354.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.13+8
> >> >> java/11.0.13 scala/2.12.7 vendor/Oracle_Corporation, )
> >> >>
> >> >> 2022-05-11 23:29:00,815 DEBUG com.amazonaws.request [] - Received
> >> >> error response: com.amazonaws.services.s3.model.AmazonS3Exception:
> Not
> >> >> Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not
> >> >> Found; ......)
> >> >>
> >> >> The key does in fact exist. How can I go about resolving this?
> >> >>
> >> >> --
> >> >> Cheers,
> >> >> Aeden
> >> >>
> >> >> GitHub: https://github.com/aedenj
> >>
> >>
> >>
> >> --
> >> Cheers,
> >> Aeden
> >>
> >> GitHub: https://github.com/aedenj
> >> Linked In: http://www.linkedin.com/in/aedenjameson
>
>
>
> --
> Cheers,
> Aeden
>
> GitHub: https://github.com/aedenj
> Linked In: http://www.linkedin.com/in/aedenjameson
>
Re: Checkpointing - Job Manager S3 Head Requests are 404s
Posted by Aeden Jameson <ae...@gmail.com>.
Great, that all makes sense to me. Thanks again.
On Thu, May 19, 2022 at 11:42 AM David Anderson <da...@apache.org> wrote:
>
> Sure, happy to try to help.
>
> What's happening with the hadoop filesystem is that before it writes each key it checks to see if the "parent directory" exists by checking for a key with the prefix up to the last "/", and if that key isn't found it then creates empty marker files to cause of that parent directory to exist. These existence checks are S3 HEAD requests. None of this is helpful in the case of Flink checkpointing.
>
> And yes, while Presto avoids this unnecessary overhead, entropy injection remains an important tool for improving scalability. Where you'll run into quotas and rate limits depends, of course, on factors like the number of stateful tasks in your job(s), the parallelism, and the checkpointing interval.
>
> On Thu, May 19, 2022 at 8:04 PM Aeden Jameson <ae...@gmail.com> wrote:
>>
>> Thanks for the response David. That's the conclusion I came to as
>> well. The Hadoop plugin behavior doesn't appear to reflect more
>> recent changes to S3 like strong read-after-write consistency,
>> https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/
>> .
>>
>> Given the improved interaction of the Presto plugin with S3 I would
>> conclude that this increases the size of the cluster that would
>> require entropy injection, yes? But that it doesn't necessarily get
>> rid of the need because one could have a large enough cluster and say
>> a lifecycle policy that could still end up requiring entropy
>> injection.
>>
>> On Thu, May 19, 2022 at 10:24 AM David Anderson <da...@apache.org> wrote:
>> >
>> > Aeden, this is probably happening because you are using the Hadoop implementation of S3.
>> >
>> > The Hadoop S3 filesystem tries to imitate a filesystem on top of S3. In so doing it makes a lot of HEAD requests. These are expensive, and they violate read-after-create visibility, which is what you seem to be experiencing. By contrast, the Presto S3 implementation doesn't do the same (harmful in this case) magic, and simply does PUT/GET operations. Because that's all Flink needs to checkpointing, this works much better.
>> >
>> > Best,
>> > David
>> >
>> > On Thu, May 12, 2022 at 1:53 AM Aeden Jameson <ae...@gmail.com> wrote:
>> >>
>> >> We're using S3 to store checkpoints. They are taken every minute. I'm
>> >> seeing a large number of 404 responses from S3 being generated by the
>> >> job manager. The order of the entries in the debugging log would imply
>> >> that it's a result of a HEAD request to a key. For example all the
>> >> incidents look like this,
>> >>
>> >>
>> >> 2022-05-11 23:29:00,804 DEBUG com.amazonaws.request [] - Sending
>> >> Request: HEAD https://[MY-BUCKET].s3.amazonaws.com
>> >> /[MY_JOB]/checkpoints/5f4d6923883a1702b206f978fa3637a3/ Headers:
>> >> (amz-sdk-invocation-id: XXXXX, Content-Type: application/octet-stream,
>> >> User-Agent: Hadoop 3.1.0, aws-sdk-java/1.11.788
>> >> Linux/5.4.181-99.354.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.13+8
>> >> java/11.0.13 scala/2.12.7 vendor/Oracle_Corporation, )
>> >>
>> >> 2022-05-11 23:29:00,815 DEBUG com.amazonaws.request [] - Received
>> >> error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not
>> >> Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not
>> >> Found; ......)
>> >>
>> >> The key does in fact exist. How can I go about resolving this?
>> >>
>> >> --
>> >> Cheers,
>> >> Aeden
>> >>
>> >> GitHub: https://github.com/aedenj
>>
>>
>>
>> --
>> Cheers,
>> Aeden
>>
>> GitHub: https://github.com/aedenj
>> Linked In: http://www.linkedin.com/in/aedenjameson
--
Cheers,
Aeden
GitHub: https://github.com/aedenj
Linked In: http://www.linkedin.com/in/aedenjameson
Re: Checkpointing - Job Manager S3 Head Requests are 404s
Posted by David Anderson <da...@apache.org>.
Sure, happy to try to help.
What's happening with the hadoop filesystem is that before it writes each
key it checks to see if the "parent directory" exists by checking for a key
with the prefix up to the last "/", and if that key isn't found it then
creates empty marker files to cause of that parent directory to exist.
These existence checks are S3 HEAD requests. None of this is helpful in the
case of Flink checkpointing.
And yes, while Presto avoids this unnecessary overhead, entropy injection
remains an important tool for improving scalability. Where you'll run into
quotas and rate limits depends, of course, on factors like the number of
stateful tasks in your job(s), the parallelism, and the checkpointing
interval.
On Thu, May 19, 2022 at 8:04 PM Aeden Jameson <ae...@gmail.com>
wrote:
> Thanks for the response David. That's the conclusion I came to as
> well. The Hadoop plugin behavior doesn't appear to reflect more
> recent changes to S3 like strong read-after-write consistency,
>
> https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/
> .
>
> Given the improved interaction of the Presto plugin with S3 I would
> conclude that this increases the size of the cluster that would
> require entropy injection, yes? But that it doesn't necessarily get
> rid of the need because one could have a large enough cluster and say
> a lifecycle policy that could still end up requiring entropy
> injection.
>
> On Thu, May 19, 2022 at 10:24 AM David Anderson <da...@apache.org>
> wrote:
> >
> > Aeden, this is probably happening because you are using the Hadoop
> implementation of S3.
> >
> > The Hadoop S3 filesystem tries to imitate a filesystem on top of S3. In
> so doing it makes a lot of HEAD requests. These are expensive, and they
> violate read-after-create visibility, which is what you seem to be
> experiencing. By contrast, the Presto S3 implementation doesn't do the same
> (harmful in this case) magic, and simply does PUT/GET operations. Because
> that's all Flink needs to checkpointing, this works much better.
> >
> > Best,
> > David
> >
> > On Thu, May 12, 2022 at 1:53 AM Aeden Jameson <ae...@gmail.com>
> wrote:
> >>
> >> We're using S3 to store checkpoints. They are taken every minute. I'm
> >> seeing a large number of 404 responses from S3 being generated by the
> >> job manager. The order of the entries in the debugging log would imply
> >> that it's a result of a HEAD request to a key. For example all the
> >> incidents look like this,
> >>
> >>
> >> 2022-05-11 23:29:00,804 DEBUG com.amazonaws.request [] - Sending
> >> Request: HEAD https://[MY-BUCKET].s3.amazonaws.com
> >> /[MY_JOB]/checkpoints/5f4d6923883a1702b206f978fa3637a3/ Headers:
> >> (amz-sdk-invocation-id: XXXXX, Content-Type: application/octet-stream,
> >> User-Agent: Hadoop 3.1.0, aws-sdk-java/1.11.788
> >> Linux/5.4.181-99.354.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.13+8
> >> java/11.0.13 scala/2.12.7 vendor/Oracle_Corporation, )
> >>
> >> 2022-05-11 23:29:00,815 DEBUG com.amazonaws.request [] - Received
> >> error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not
> >> Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not
> >> Found; ......)
> >>
> >> The key does in fact exist. How can I go about resolving this?
> >>
> >> --
> >> Cheers,
> >> Aeden
> >>
> >> GitHub: https://github.com/aedenj
>
>
>
> --
> Cheers,
> Aeden
>
> GitHub: https://github.com/aedenj
> Linked In: http://www.linkedin.com/in/aedenjameson
>
Re: Checkpointing - Job Manager S3 Head Requests are 404s
Posted by Aeden Jameson <ae...@gmail.com>.
Thanks for the response David. That's the conclusion I came to as
well. The Hadoop plugin behavior doesn't appear to reflect more
recent changes to S3 like strong read-after-write consistency,
https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/
.
Given the improved interaction of the Presto plugin with S3 I would
conclude that this increases the size of the cluster that would
require entropy injection, yes? But that it doesn't necessarily get
rid of the need because one could have a large enough cluster and say
a lifecycle policy that could still end up requiring entropy
injection.
On Thu, May 19, 2022 at 10:24 AM David Anderson <da...@apache.org> wrote:
>
> Aeden, this is probably happening because you are using the Hadoop implementation of S3.
>
> The Hadoop S3 filesystem tries to imitate a filesystem on top of S3. In so doing it makes a lot of HEAD requests. These are expensive, and they violate read-after-create visibility, which is what you seem to be experiencing. By contrast, the Presto S3 implementation doesn't do the same (harmful in this case) magic, and simply does PUT/GET operations. Because that's all Flink needs to checkpointing, this works much better.
>
> Best,
> David
>
> On Thu, May 12, 2022 at 1:53 AM Aeden Jameson <ae...@gmail.com> wrote:
>>
>> We're using S3 to store checkpoints. They are taken every minute. I'm
>> seeing a large number of 404 responses from S3 being generated by the
>> job manager. The order of the entries in the debugging log would imply
>> that it's a result of a HEAD request to a key. For example all the
>> incidents look like this,
>>
>>
>> 2022-05-11 23:29:00,804 DEBUG com.amazonaws.request [] - Sending
>> Request: HEAD https://[MY-BUCKET].s3.amazonaws.com
>> /[MY_JOB]/checkpoints/5f4d6923883a1702b206f978fa3637a3/ Headers:
>> (amz-sdk-invocation-id: XXXXX, Content-Type: application/octet-stream,
>> User-Agent: Hadoop 3.1.0, aws-sdk-java/1.11.788
>> Linux/5.4.181-99.354.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.13+8
>> java/11.0.13 scala/2.12.7 vendor/Oracle_Corporation, )
>>
>> 2022-05-11 23:29:00,815 DEBUG com.amazonaws.request [] - Received
>> error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not
>> Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not
>> Found; ......)
>>
>> The key does in fact exist. How can I go about resolving this?
>>
>> --
>> Cheers,
>> Aeden
>>
>> GitHub: https://github.com/aedenj
--
Cheers,
Aeden
GitHub: https://github.com/aedenj
Linked In: http://www.linkedin.com/in/aedenjameson
Re: Checkpointing - Job Manager S3 Head Requests are 404s
Posted by David Anderson <da...@apache.org>.
Aeden, this is probably happening because you are using the Hadoop
implementation of S3.
The Hadoop S3 filesystem tries to imitate a filesystem on top of S3. In so
doing it makes a lot of HEAD requests. These are expensive, and they
violate read-after-create visibility, which is what you seem to be
experiencing. By contrast, the Presto S3 implementation doesn't do the same
(harmful in this case) magic, and simply does PUT/GET operations. Because
that's all Flink needs to checkpointing, this works much better.
Best,
David
On Thu, May 12, 2022 at 1:53 AM Aeden Jameson <ae...@gmail.com>
wrote:
> We're using S3 to store checkpoints. They are taken every minute. I'm
> seeing a large number of 404 responses from S3 being generated by the
> job manager. The order of the entries in the debugging log would imply
> that it's a result of a HEAD request to a key. For example all the
> incidents look like this,
>
>
> 2022-05-11 23:29:00,804 DEBUG com.amazonaws.request [] - Sending
> Request: HEAD https://[MY-BUCKET].s3.amazonaws.com
> /[MY_JOB]/checkpoints/5f4d6923883a1702b206f978fa3637a3/ Headers:
> (amz-sdk-invocation-id: XXXXX, Content-Type: application/octet-stream,
> User-Agent: Hadoop 3.1.0, aws-sdk-java/1.11.788
> Linux/5.4.181-99.354.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.13+8
> java/11.0.13 scala/2.12.7 vendor/Oracle_Corporation, )
>
> 2022-05-11 23:29:00,815 DEBUG com.amazonaws.request [] - Received
> error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not
> Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not
> Found; ......)
>
> The key does in fact exist. How can I go about resolving this?
>
> --
> Cheers,
> Aeden
>
> GitHub: https://github.com/aedenj
>