You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishwas Siravara <vs...@gmail.com> on 2019/07/19 00:38:56 UTC
Flink s3 wire log
Here is my wire log while trying to checkpoint to ecs S3. I see the request
got a 404 , does this mean that it can't find the folder *checkpoints . *Since
s3 does not have folders, what should I put there ? Thanks so much for all
the help that you guys have provided so far. Really appreciate it.
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.com.amazonaws.internal.SdkSSLSocket
- created: vishwas.test1.SU73ECSG2P1d.VISA.COM/192.168.38.48:9021
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator
- Connection established 10.211.120.115:44746<->192.168.38.48:9021
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.conn.DefaultManagedHttpClientConnection
- http-outgoing-15: set socket timeout to 200000
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.MainClientExec
- Executing request HEAD
/checkpoints/6a17b04819b5d567afd1376faa174e1d HTTP/1.1
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.impl.execchain.MainClientExec
- Proxy auth state: UNCHALLENGED
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.headers -
http-outgoing-15 >> HEAD /checkpoints/6a17b04819b5d567afd1376faa174e1d
HTTP/1.1
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.headers -
http-outgoing-15 >> Host: vishwas.test1.SU73ECSG2P1d.VISA.COM:9021
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.headers -
http-outgoing-15 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.headers -
http-outgoing-15 >> Authorization: AWS4-HMAC-SHA256
Credential=vdna_np_user/20190719/us-east-1/s3/aws4_request,
SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date,
Signature=bf5910dae4c760b1ebd80307e51516f26a6fe5e8adfbd5fcb0a3f74da7c0891e
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.headers -
http-outgoing-15 >> X-Amz-Date: 20190719T002936Z
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.headers -
http-outgoing-15 >> User-Agent: Hadoop 3.1.0, aws-sdk-java/1.11.271
Linux/3.10.0-957.21.3.el7.x86_64
Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11 java/1.8.0_131
scala/2.11.12
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.headers -
http-outgoing-15 >> amz-sdk-invocation-id:
c5d5c46a-ff2d-1ff2-6b7a-f43ab9b78e5d
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.headers -
http-outgoing-15 >> amz-sdk-retry: 0/0/500
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.headers -
http-outgoing-15 >> Content-Type: application/octet-stream
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.headers -
http-outgoing-15 >> Connection: Keep-Alive
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "HEAD
/checkpoints/6a17b04819b5d567afd1376faa174e1d HTTP/1.1[\r][\n]"
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "Host:
vishwas.test1.SU73ECSG2P1d.VISA.COM:9021[\r][\n]"
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "Authorization: AWS4-HMAC-SHA256
Credential=vdna_np_user/20190719/us-east-1/s3/aws4_request,
SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date,
Signature=bf5910dae4c760b1ebd80307e51516f26a6fe5e8adfbd5fcb0a3f74da7c0891e[\r][\n]"
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "X-Amz-Date: 20190719T002936Z[\r][\n]"
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "User-Agent: Hadoop 3.1.0, aws-sdk-java/1.11.271
Linux/3.10.0-957.21.3.el7.x86_64
Java_HotSpot(TM)_64-Bit_Server_VM/25.131-b11 java/1.8.0_131
scala/2.11.12[\r][\n]"
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "amz-sdk-invocation-id:
c5d5c46a-ff2d-1ff2-6b7a-f43ab9b78e5d[\r][\n]"
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "amz-sdk-retry: 0/0/500[\r][\n]"
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "Content-Type: application/octet-stream[\r][\n]"
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "Connection: Keep-Alive[\r][\n]"
2019-07-19 00:29:36,323 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 >> "[\r][\n]"
2019-07-19 00:29:36,356 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 << "HTTP/1.1 404 Not Found[\r][\n]"
2019-07-19 00:29:36,356 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 << "Date: Fri, 19 Jul 2019 00:29:36 GMT[\r][\n]"
2019-07-19 00:29:36,356 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 << "Date: Fri, 19 Jul 2019 00:29:36 GMT[\r][\n]"
2019-07-19 00:29:36,356 DEBUG
org.apache.flink.fs.s3base.shaded.org.apache.http.wire -
http-outgoing-15 << "x-amz-id-2: [\r][\n]"