You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Fabian Wollert <fa...@zalando.de> on 2018/05/30 14:08:38 UTC
Flink and AWS S3 integration: java.lang.NullPointerException: null
uri host
Hi, I'm trying to set up Checkpoints for Flink Jobs with S3 as a filesystem
backend. I configured the following:
state.backend=filesystem
state.backend.fs.checkpointdir=s3:///mybucket/
state.checkpoints.dir=s3:///mybucket/
state.checkpoints.num-retained=3
I also copied the flink-s3-fs-hadoop-1.5.0.jar into the lib folder.
I get now though the following error message:
Caused by: java.lang.NullPointerException: null uri host.
at java.util.Objects.requireNonNull(Objects.java:228)
at
org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:65)
at
org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:165)
at
org.apache.flink.fs.s3hadoop.S3FileSystemFactory.create(S3FileSystemFactory.java:133)
I tried to dig deeper into the source code, but struggled to find
- what is meant with this URI
- where to configure it
Can anybody give some advice how to set up the S3 Backend with the new
shaded lib jar?
Thanks in advance
--
*Fabian WollertZalando SE*
E-Mail: fabian.wollert@zalando.de
Tamara-Danz-Straße 1
10243 Berlin
Fax: +49 (0)30 2759 46 93
E-mail: legalnotice@zalando.co.uk
Notifications of major holdings (Sec. 33, 38, 39 WpHG): +49 (0)30
2000889349
Management Board:
Robert Gentz, David Schneider, Rubin Ritter
Chairman of the Supervisory Board:
Lothar Lanz
Person responsible for providing the contents of Zalando SE acc. to Art. 55
RStV [Interstate Broadcasting Agreement]: Rubin Ritter
Registered at the Local Court Charlottenburg Berlin, HRB 158855 B
VAT registration number: DE 260543043
Re: Flink and AWS S3 integration: java.lang.NullPointerException:
null uri host
Posted by Fabian Wollert <fa...@zalando.de>.
i solved it by myself, with the help of some debugging. i used
s3:///mybucket/ but it needs to be s3://mybucket/some_folder ... 2 slashes,
and also a folder needs to be specified ...
--
*Fabian WollertZalando SE*
E-Mail: fabian.wollert@zalando.de
Location: ZMAP <ht...@zalando.de>
Am Do., 31. Mai 2018 um 09:31 Uhr schrieb Fabian Wollert <fabian@zalando.de
>:
> I'm running it in docker on EC2, cant use EMR ... yes i followed those
> instructions.
>
> Cheers
>
> --
>
>
> *Fabian WollertZalando SE*
>
> E-Mail: fabian.wollert@zalando.de
>
>
> Am Do., 31. Mai 2018 um 03:07 Uhr schrieb Bowen Li <bo...@gmail.com>:
>
>> Did you run Flink on AWS EMR or somewhere else? Have you read and
>> followed instructions on
>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#amazon-web-services-aws
>> ?
>>
>>
>>
>> On Wed, May 30, 2018 at 7:08 AM, Fabian Wollert <fa...@zalando.de>
>> wrote:
>>
>>> Hi, I'm trying to set up Checkpoints for Flink Jobs with S3 as a
>>> filesystem backend. I configured the following:
>>>
>>> state.backend=filesystem
>>> state.backend.fs.checkpointdir=s3:///mybucket/
>>> state.checkpoints.dir=s3:///mybucket/
>>> state.checkpoints.num-retained=3
>>>
>>> I also copied the flink-s3-fs-hadoop-1.5.0.jar into the lib folder.
>>>
>>> I get now though the following error message:
>>>
>>> Caused by: java.lang.NullPointerException: null uri host.
>>> at java.util.Objects.requireNonNull(Objects.java:228)
>>> at
>>> org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:65)
>>> at
>>> org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:165)
>>> at
>>> org.apache.flink.fs.s3hadoop.S3FileSystemFactory.create(S3FileSystemFactory.java:133)
>>>
>>> I tried to dig deeper into the source code, but struggled to find
>>>
>>> - what is meant with this URI
>>> - where to configure it
>>>
>>> Can anybody give some advice how to set up the S3 Backend with the new
>>> shaded lib jar?
>>>
>>> Thanks in advance
>>> --
>>>
>>>
>>> *Fabian WollertZalando SE*
>>>
>>> E-Mail: fabian.wollert@zalando.de
>>>
>>> Tamara-Danz-Straße 1
>>> <https://maps.google.com/?q=Tamara-Danz-Stra%C3%9Fe+1+10243+Berlin&entry=gmail&source=g>
>>> 10243 Berlin
>>> <https://maps.google.com/?q=Tamara-Danz-Stra%C3%9Fe+1+10243+Berlin&entry=gmail&source=g>
>>> Fax: +49 (0)30 2759 46 93
>>> E-mail: legalnotice@zalando.co.uk
>>> Notifications of major holdings (Sec. 33, 38, 39 WpHG): +49 (0)30
>>> 2000889349
>>>
>>> Management Board:
>>> Robert Gentz, David Schneider, Rubin Ritter
>>>
>>> Chairman of the Supervisory Board:
>>> Lothar Lanz
>>>
>>> Person responsible for providing the contents of Zalando SE acc. to Art.
>>> 55 RStV [Interstate Broadcasting Agreement]: Rubin Ritter
>>> Registered at the Local Court Charlottenburg Berlin, HRB 158855 B
>>> VAT registration number: DE 260543043
>>>
>>
>>
Re: Flink and AWS S3 integration: java.lang.NullPointerException:
null uri host
Posted by Fabian Wollert <fa...@zalando.de>.
I'm running it in docker on EC2, cant use EMR ... yes i followed those
instructions.
Cheers
--
*Fabian WollertZalando SE*
E-Mail: fabian.wollert@zalando.de
Am Do., 31. Mai 2018 um 03:07 Uhr schrieb Bowen Li <bo...@gmail.com>:
> Did you run Flink on AWS EMR or somewhere else? Have you read and followed
> instructions on
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#amazon-web-services-aws
> ?
>
>
>
> On Wed, May 30, 2018 at 7:08 AM, Fabian Wollert <fa...@zalando.de> wrote:
>
>> Hi, I'm trying to set up Checkpoints for Flink Jobs with S3 as a
>> filesystem backend. I configured the following:
>>
>> state.backend=filesystem
>> state.backend.fs.checkpointdir=s3:///mybucket/
>> state.checkpoints.dir=s3:///mybucket/
>> state.checkpoints.num-retained=3
>>
>> I also copied the flink-s3-fs-hadoop-1.5.0.jar into the lib folder.
>>
>> I get now though the following error message:
>>
>> Caused by: java.lang.NullPointerException: null uri host.
>> at java.util.Objects.requireNonNull(Objects.java:228)
>> at
>> org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:65)
>> at
>> org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:165)
>> at
>> org.apache.flink.fs.s3hadoop.S3FileSystemFactory.create(S3FileSystemFactory.java:133)
>>
>> I tried to dig deeper into the source code, but struggled to find
>>
>> - what is meant with this URI
>> - where to configure it
>>
>> Can anybody give some advice how to set up the S3 Backend with the new
>> shaded lib jar?
>>
>> Thanks in advance
>> --
>>
>>
>> *Fabian WollertZalando SE*
>>
>> E-Mail: fabian.wollert@zalando.de
>>
>> Tamara-Danz-Straße 1
>> <https://maps.google.com/?q=Tamara-Danz-Stra%C3%9Fe+1+10243+Berlin&entry=gmail&source=g>
>> 10243 Berlin
>> <https://maps.google.com/?q=Tamara-Danz-Stra%C3%9Fe+1+10243+Berlin&entry=gmail&source=g>
>> Fax: +49 (0)30 2759 46 93
>> E-mail: legalnotice@zalando.co.uk
>> Notifications of major holdings (Sec. 33, 38, 39 WpHG): +49 (0)30
>> 2000889349
>>
>> Management Board:
>> Robert Gentz, David Schneider, Rubin Ritter
>>
>> Chairman of the Supervisory Board:
>> Lothar Lanz
>>
>> Person responsible for providing the contents of Zalando SE acc. to Art.
>> 55 RStV [Interstate Broadcasting Agreement]: Rubin Ritter
>> Registered at the Local Court Charlottenburg Berlin, HRB 158855 B
>> VAT registration number: DE 260543043
>>
>
>
Re: Flink and AWS S3 integration: java.lang.NullPointerException:
null uri host
Posted by Bowen Li <bo...@gmail.com>.
Did you run Flink on AWS EMR or somewhere else? Have you read and followed
instructions on
https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/deployment/aws.html#amazon-web-services-aws
?
On Wed, May 30, 2018 at 7:08 AM, Fabian Wollert <fa...@zalando.de> wrote:
> Hi, I'm trying to set up Checkpoints for Flink Jobs with S3 as a
> filesystem backend. I configured the following:
>
> state.backend=filesystem
> state.backend.fs.checkpointdir=s3:///mybucket/
> state.checkpoints.dir=s3:///mybucket/
> state.checkpoints.num-retained=3
>
> I also copied the flink-s3-fs-hadoop-1.5.0.jar into the lib folder.
>
> I get now though the following error message:
>
> Caused by: java.lang.NullPointerException: null uri host.
> at java.util.Objects.requireNonNull(Objects.java:228)
> at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.
> s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:65)
> at org.apache.flink.fs.s3hadoop.shaded.org.apache.hadoop.fs.
> s3a.S3AFileSystem.initialize(S3AFileSystem.java:165)
> at org.apache.flink.fs.s3hadoop.S3FileSystemFactory.create(
> S3FileSystemFactory.java:133)
>
> I tried to dig deeper into the source code, but struggled to find
>
> - what is meant with this URI
> - where to configure it
>
> Can anybody give some advice how to set up the S3 Backend with the new
> shaded lib jar?
>
> Thanks in advance
> --
>
>
> *Fabian WollertZalando SE*
>
> E-Mail: fabian.wollert@zalando.de
>
> Tamara-Danz-Straße 1
> <https://maps.google.com/?q=Tamara-Danz-Stra%C3%9Fe+1+10243+Berlin&entry=gmail&source=g>
> 10243 Berlin
> <https://maps.google.com/?q=Tamara-Danz-Stra%C3%9Fe+1+10243+Berlin&entry=gmail&source=g>
> Fax: +49 (0)30 2759 46 93
> E-mail: legalnotice@zalando.co.uk
> Notifications of major holdings (Sec. 33, 38, 39 WpHG): +49 (0)30
> 2000889349
>
> Management Board:
> Robert Gentz, David Schneider, Rubin Ritter
>
> Chairman of the Supervisory Board:
> Lothar Lanz
>
> Person responsible for providing the contents of Zalando SE acc. to Art.
> 55 RStV [Interstate Broadcasting Agreement]: Rubin Ritter
> Registered at the Local Court Charlottenburg Berlin, HRB 158855 B
> VAT registration number: DE 260543043
>