You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Bowen Li (JIRA)" <ji...@apache.org> on 2017/08/03 21:21:00 UTC
[jira] [Created] (FLINK-7365) warning of attempt to override final
parameter: fs.s3.buffer.dir
Bowen Li created FLINK-7365:
-------------------------------
Summary: warning of attempt to override final parameter: fs.s3.buffer.dir
Key: FLINK-7365
URL: https://issues.apache.org/jira/browse/FLINK-7365
Project: Flink
Issue Type: Bug
Reporter: Bowen Li
I'm seeing hundreds of line of the following log in my JobManager log file:
{code:java}
2017-08-03 19:48:45,330 WARN org.apache.hadoop.conf.Configuration - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir; Ignoring.
2017-08-03 19:48:45,485 WARN org.apache.hadoop.conf.Configuration - /etc/hadoop/conf/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir; Ignoring.
2017-08-03 19:48:45,486 WARN org.apache.hadoop.conf.Configuration - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir; Ignoring.
2017-08-03 19:48:45,626 WARN org.apache.hadoop.conf.Configuration - /etc/hadoop/conf/core-site.xml:an attempt to override final parameter: fs.s3.buffer.dir; Ignoring
......
{code}
Info of my Flink cluster:
- Running on EMR with emr-5.6.0
- Using FSStateBackend, writing checkpointing data files to s3
- Configured s3 with S3AFileSystem according to https://ci.apache.org/projects/flink/flink-docs-release-1.4/setup/aws.html#set-s3-filesystem
- AWS forbids resetting 'fs.s3.buffer.dir' value (it has a <final> tag on this property in core-site.xml), so I set 'fs.s3a.buffer.dir' as '/tmp'
Here's my core-site.xml file:
{code:java}
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
<configuration>
<property>
<name>fs.s3.buffer.dir</name>
<value>/mnt/s3,/mnt1/s3</value>
<final>true</final>
</property>
<property>
<name>fs.s3.impl</name>
<value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>
<property>
<name>fs.s3n.impl</name>
<value>com.amazon.ws.emr.hadoop.fs.EmrFileSystem</value>
</property>
<property>
<name>ipc.client.connect.max.retries.on.timeouts</name>
<value>5</value>
</property>
<property>
<name>hadoop.security.key.default.bitlength</name>
<value>256</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/mnt/var/lib/hadoop/tmp</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>65536</value>
</property>
<property>
<name>fs.AbstractFileSystem.s3.impl</name>
<value>org.apache.hadoop.fs.s3.EMRFSDelegate</value>
</property>
<property>
<name>fs.s3a.buffer.dir</name>
<value>/tmp</value>
</property>
<property>
<name>fs.s3bfs.impl</name>
<value>org.apache.hadoop.fs.s3.S3FileSystem</value>
</property>
</configuration>
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)