[jira] [Commented] (FLINK-38210) s5cmd somtimes fails due to path starting with s3p://

["Zackary Naas (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-38210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18044695#comment-18044695 ] 

Zackary Naas commented on FLINK-38210:
--------------------------------------

I think I see how to fix this - seems like the `convertToSpells` [function|https://github.com/apache/flink/blob/7ba97419d200adef672b67c52c2b5a79f277cfda/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#L304] just needs to account for URIs that start with s3p or s3a and convert them to s3. I can open a PR for this today.

> s5cmd somtimes fails due to path starting with s3p://
> -----------------------------------------------------
>
>                 Key: FLINK-38210
>                 URL: https://issues.apache.org/jira/browse/FLINK-38210
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Checkpointing
>    Affects Versions: 2.1.0
>            Reporter: Grzegorz Liter
>            Priority: Major
>
> s5cmd fails to startup job due to path starting with s3p:// which is recommended way when both Hadoop and Presto implementations are in use.
>  
> {code:java}
> Caused by: java.io.IOException: Failed to cast s5cmd spell [/opt/flink/opt/s5cmd/s5cmd -r 0 run] [exit code = 1] [cfg: S5CmdConfiguration{path='/opt/flink/opt/s5cmd/s5cmd', args=[-r, 0], accessArtifact='null', secretArtifact='null', endpoint='null'}] maybe due to:ERROR "cp s3p://bucket/checkpoints/9d2739a593435f30a044965427d3d53e/shared/6c726cc2-f51c-42e3-8647-f6de0e32040d /tmp/tm_dema-flink-stream-processor-taskmanager-1-2/tmp/job_9d2739a593435f30a044965427d3d53e_op_WindowOperator_4ddf880b77ad626da96b1c364685b172__1_6__uuid_96cd1a7e-52ba-46f9-8c3d-4674f4d573f5/942ab49c-d8ae-4d3e-b5df-0d0c700705c9/000019.sst": s3 url should start with "s3://" {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)