You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Matt Fysh <ma...@unweb.io> on 2022/10/20 07:52:05 UTC
Unable to get Flink S3 plugin to work
Hi Zeppers,
I'm running Flink inside the Zeppelin 0.10 docker image, and having
difficulty writing a stream to S3. I've copied the plugin inside the Flink
plugin directory, but I think it also requires the Service Loader to be
configured according to this StackOverflow question
<https://stackoverflow.com/questions/64115627/flink-1-11-2-cant-find-implementation-for-s3-despite-correct-plugins-being>
I was wondering how I can get this to work when running Flink inside of
Zeppelin? Given that the jobs we write in notebooks don't have a permanent
JAR, I'm at a loss to understand where I should palace this META-INF folder
with the services directory inside
Thanks,
Matt
Re: Unable to get Flink S3 plugin to work
Posted by Jeff Zhang <zj...@gmail.com>.
Could you try to copy the plugin jar into FLINK_HOME/lib?
On Fri, Oct 21, 2022 at 4:59 AM Matt Fysh <ma...@unweb.io> wrote:
> Hi Jeff,
>
> I created a small reproduction repo here:
> https://github.com/mattfysh/zepp-flink-s3 and the error I'm seeing is:
>
> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not
> find a file system implementation for scheme 's3a'
>
>
> I'm using version 1.13.2 to match AWS where I hope to deploy my final app,
> and I've included the steps to copy the plugin into the plugins directory
> in the install.sh script
>
> I see this same error regardless of whether I try using the Hadoop or
> Presto plugins, it seems that the plugin is not being loaded by the service
> loader, so there may be some additional config required?
>
> Thanks,
> Matt
>
> On Thu, 20 Oct 2022 at 18:43, Jeff Zhang <zj...@gmail.com> wrote:
>
>> What exact error do you mean? Even if you run flink inside zeppelin, you
>> still need to configure FLINK_HOME, for any jars, you need to put under
>> FLINK_HOME I believe.
>>
>>
>> On Thu, Oct 20, 2022 at 3:52 PM Matt Fysh <ma...@unweb.io> wrote:
>>
>>> Hi Zeppers,
>>>
>>> I'm running Flink inside the Zeppelin 0.10 docker image, and having
>>> difficulty writing a stream to S3. I've copied the plugin inside the Flink
>>> plugin directory, but I think it also requires the Service Loader to be
>>> configured according to this StackOverflow question
>>> <https://stackoverflow.com/questions/64115627/flink-1-11-2-cant-find-implementation-for-s3-despite-correct-plugins-being>
>>>
>>> I was wondering how I can get this to work when running Flink inside of
>>> Zeppelin? Given that the jobs we write in notebooks don't have a permanent
>>> JAR, I'm at a loss to understand where I should palace this META-INF folder
>>> with the services directory inside
>>>
>>> Thanks,
>>> Matt
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
--
Best Regards
Jeff Zhang
Re: Unable to get Flink S3 plugin to work
Posted by Matt Fysh <ma...@unweb.io>.
Hi Jeff,
I created a small reproduction repo here:
https://github.com/mattfysh/zepp-flink-s3 and the error I'm seeing is:
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not
find a file system implementation for scheme 's3a'
I'm using version 1.13.2 to match AWS where I hope to deploy my final app,
and I've included the steps to copy the plugin into the plugins directory
in the install.sh script
I see this same error regardless of whether I try using the Hadoop or
Presto plugins, it seems that the plugin is not being loaded by the service
loader, so there may be some additional config required?
Thanks,
Matt
On Thu, 20 Oct 2022 at 18:43, Jeff Zhang <zj...@gmail.com> wrote:
> What exact error do you mean? Even if you run flink inside zeppelin, you
> still need to configure FLINK_HOME, for any jars, you need to put under
> FLINK_HOME I believe.
>
>
> On Thu, Oct 20, 2022 at 3:52 PM Matt Fysh <ma...@unweb.io> wrote:
>
>> Hi Zeppers,
>>
>> I'm running Flink inside the Zeppelin 0.10 docker image, and having
>> difficulty writing a stream to S3. I've copied the plugin inside the Flink
>> plugin directory, but I think it also requires the Service Loader to be
>> configured according to this StackOverflow question
>> <https://stackoverflow.com/questions/64115627/flink-1-11-2-cant-find-implementation-for-s3-despite-correct-plugins-being>
>>
>> I was wondering how I can get this to work when running Flink inside of
>> Zeppelin? Given that the jobs we write in notebooks don't have a permanent
>> JAR, I'm at a loss to understand where I should palace this META-INF folder
>> with the services directory inside
>>
>> Thanks,
>> Matt
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Unable to get Flink S3 plugin to work
Posted by Jeff Zhang <zj...@gmail.com>.
What exact error do you mean? Even if you run flink inside zeppelin, you
still need to configure FLINK_HOME, for any jars, you need to put under
FLINK_HOME I believe.
On Thu, Oct 20, 2022 at 3:52 PM Matt Fysh <ma...@unweb.io> wrote:
> Hi Zeppers,
>
> I'm running Flink inside the Zeppelin 0.10 docker image, and having
> difficulty writing a stream to S3. I've copied the plugin inside the Flink
> plugin directory, but I think it also requires the Service Loader to be
> configured according to this StackOverflow question
> <https://stackoverflow.com/questions/64115627/flink-1-11-2-cant-find-implementation-for-s3-despite-correct-plugins-being>
>
> I was wondering how I can get this to work when running Flink inside of
> Zeppelin? Given that the jobs we write in notebooks don't have a permanent
> JAR, I'm at a loss to understand where I should palace this META-INF folder
> with the services directory inside
>
> Thanks,
> Matt
>
--
Best Regards
Jeff Zhang