You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/08/07 09:58:10 UTC
[GitHub] [kafka] tombentley opened a new pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
tombentley opened a new pull request #9136:
URL: https://github.com/apache/kafka/pull/9136
See KIP-632.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison merged pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
mimaison merged pull request #9136:
URL: https://github.com/apache/kafka/pull/9136
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on a change in pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
mimaison commented on a change in pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#discussion_r467206725
##########
File path: clients/src/main/java/org/apache/kafka/common/config/provider/DirectoryConfigProvider.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config.provider;
+
+import org.apache.kafka.common.config.ConfigData;
+import org.apache.kafka.common.config.ConfigException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * An implementation of {@link ConfigProvider} based on a directory of files.
+ * Property keys correspond to the names of the regular (i.e. non-directory)
+ * files in a directory given by the path parameter.
+ * Property values are taken from the file contents corresponding to each key.
+ */
+public class DirectoryConfigProvider implements ConfigProvider {
+
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ @Override
+ public void configure(Map<String, ?> configs) { }
+
+ @Override
+ public void close() throws IOException { }
+
+ /**
+ * Retrieves the data contained in regular files in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path) {
+ return get(path, File::isFile);
+ }
+
+ /**
+ * Retrieves the data contained in the regular files named by {@code keys} in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @param keys the keys whose values will be retrieved.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path, Set<String> keys) {
+ return get(path, pathname ->
+ pathname.isFile()
+ && keys.contains(pathname.getName()));
+ }
+
+ private ConfigData get(String path, FileFilter fileFilter) {
+ Map<String, String> map = new HashMap<>();
+ if (path != null && !path.isEmpty()) {
+ File dir = new File(path);
+ if (!dir.isDirectory()) {
+ log.warn("The path {} is not a directory", path);
+ } else {
+ for (File file : dir.listFiles(fileFilter)) {
Review comment:
Should we use `java.nio.Files` here instead? That will also give us an Exception if an IO error happens instead of `null`.
##########
File path: clients/src/main/java/org/apache/kafka/common/config/provider/DirectoryConfigProvider.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config.provider;
+
+import org.apache.kafka.common.config.ConfigData;
+import org.apache.kafka.common.config.ConfigException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * An implementation of {@link ConfigProvider} based on a directory of files.
+ * Property keys correspond to the names of the regular (i.e. non-directory)
+ * files in a directory given by the path parameter.
+ * Property values are taken from the file contents corresponding to each key.
+ */
+public class DirectoryConfigProvider implements ConfigProvider {
+
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ @Override
+ public void configure(Map<String, ?> configs) { }
+
+ @Override
+ public void close() throws IOException { }
+
+ /**
+ * Retrieves the data contained in regular files in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path) {
+ return get(path, File::isFile);
+ }
+
+ /**
+ * Retrieves the data contained in the regular files named by {@code keys} in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @param keys the keys whose values will be retrieved.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path, Set<String> keys) {
+ return get(path, pathname ->
+ pathname.isFile()
+ && keys.contains(pathname.getName()));
+ }
+
+ private ConfigData get(String path, FileFilter fileFilter) {
Review comment:
If we make `log` static, this can be `static` too
##########
File path: clients/src/main/java/org/apache/kafka/common/config/provider/DirectoryConfigProvider.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config.provider;
+
+import org.apache.kafka.common.config.ConfigData;
+import org.apache.kafka.common.config.ConfigException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * An implementation of {@link ConfigProvider} based on a directory of files.
+ * Property keys correspond to the names of the regular (i.e. non-directory)
+ * files in a directory given by the path parameter.
+ * Property values are taken from the file contents corresponding to each key.
+ */
+public class DirectoryConfigProvider implements ConfigProvider {
+
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ @Override
+ public void configure(Map<String, ?> configs) { }
+
+ @Override
+ public void close() throws IOException { }
+
+ /**
+ * Retrieves the data contained in regular files in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path) {
+ return get(path, File::isFile);
+ }
+
+ /**
+ * Retrieves the data contained in the regular files named by {@code keys} in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @param keys the keys whose values will be retrieved.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path, Set<String> keys) {
+ return get(path, pathname ->
+ pathname.isFile()
+ && keys.contains(pathname.getName()));
+ }
+
+ private ConfigData get(String path, FileFilter fileFilter) {
+ Map<String, String> map = new HashMap<>();
+ if (path != null && !path.isEmpty()) {
+ File dir = new File(path);
+ if (!dir.isDirectory()) {
+ log.warn("The path {} is not a directory", path);
+ } else {
+ for (File file : dir.listFiles(fileFilter)) {
+ try {
+ map.put(file.getName(), read(file.toPath()));
+ } catch (IOException e) {
+ throw new ConfigException("Could not read file " + file.getAbsolutePath() + " for property " + file.getName(), e);
+ }
+ }
+ }
+ }
+ return new ConfigData(map);
+ }
+
+ private String read(Path path) throws IOException {
Review comment:
This can be `static`
##########
File path: clients/src/test/java/org/apache/kafka/common/config/provider/DirectoryConfigProviderTest.java
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config.provider;
+
+import org.apache.kafka.common.config.ConfigData;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.Collections;
+import java.util.Locale;
+import java.util.Set;
+
+import static java.util.Arrays.asList;
+import static org.apache.kafka.test.TestUtils.toSet;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class DirectoryConfigProviderTest {
+
+ private DirectoryConfigProvider provider;
+ private File parent;
+ private File dir;
+ private File bar;
+ private File foo;
+ private File subdir;
+ private File subdirFile;
+ private File siblingDir;
+ private File siblingDirFile;
+ private File siblingFile;
+
+ private static File writeFile(File file) throws IOException {
+ Files.write(file.toPath(), file.getName().toUpperCase(Locale.ENGLISH).getBytes(StandardCharsets.UTF_8));
+ return file;
+ }
+
+ @Before
+ public void setup() throws IOException {
+ provider = new DirectoryConfigProvider();
+ provider.configure(Collections.emptyMap());
+ parent = TestUtils.tempDirectory();
Review comment:
Should we delete anything that gets created in this directory?
##########
File path: clients/src/main/java/org/apache/kafka/common/config/provider/DirectoryConfigProvider.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config.provider;
+
+import org.apache.kafka.common.config.ConfigData;
+import org.apache.kafka.common.config.ConfigException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * An implementation of {@link ConfigProvider} based on a directory of files.
+ * Property keys correspond to the names of the regular (i.e. non-directory)
+ * files in a directory given by the path parameter.
+ * Property values are taken from the file contents corresponding to each key.
+ */
+public class DirectoryConfigProvider implements ConfigProvider {
+
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ @Override
+ public void configure(Map<String, ?> configs) { }
+
+ @Override
+ public void close() throws IOException { }
+
+ /**
+ * Retrieves the data contained in regular files in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path) {
+ return get(path, File::isFile);
+ }
+
+ /**
+ * Retrieves the data contained in the regular files named by {@code keys} in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @param keys the keys whose values will be retrieved.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path, Set<String> keys) {
+ return get(path, pathname ->
+ pathname.isFile()
+ && keys.contains(pathname.getName()));
+ }
+
+ private ConfigData get(String path, FileFilter fileFilter) {
+ Map<String, String> map = new HashMap<>();
+ if (path != null && !path.isEmpty()) {
+ File dir = new File(path);
+ if (!dir.isDirectory()) {
+ log.warn("The path {} is not a directory", path);
+ } else {
+ for (File file : dir.listFiles(fileFilter)) {
Review comment:
Actually it's worth checking if we can fully switch to `java.nio` classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tombentley commented on a change in pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
tombentley commented on a change in pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#discussion_r467984879
##########
File path: clients/src/main/java/org/apache/kafka/common/config/provider/DirectoryConfigProvider.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config.provider;
+
+import org.apache.kafka.common.config.ConfigData;
+import org.apache.kafka.common.config.ConfigException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * An implementation of {@link ConfigProvider} based on a directory of files.
+ * Property keys correspond to the names of the regular (i.e. non-directory)
+ * files in a directory given by the path parameter.
+ * Property values are taken from the file contents corresponding to each key.
+ */
+public class DirectoryConfigProvider implements ConfigProvider {
+
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ @Override
+ public void configure(Map<String, ?> configs) { }
+
+ @Override
+ public void close() throws IOException { }
+
+ /**
+ * Retrieves the data contained in regular files in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path) {
+ return get(path, File::isFile);
+ }
+
+ /**
+ * Retrieves the data contained in the regular files named by {@code keys} in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @param keys the keys whose values will be retrieved.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path, Set<String> keys) {
+ return get(path, pathname ->
+ pathname.isFile()
+ && keys.contains(pathname.getName()));
+ }
+
+ private ConfigData get(String path, FileFilter fileFilter) {
+ Map<String, String> map = new HashMap<>();
+ if (path != null && !path.isEmpty()) {
+ File dir = new File(path);
+ if (!dir.isDirectory()) {
+ log.warn("The path {} is not a directory", path);
+ } else {
+ for (File file : dir.listFiles(fileFilter)) {
Review comment:
That's a good point about null being used for the error case as well as the directory case. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
mimaison commented on pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#issuecomment-677487738
@tombentley The change looks good to me but there's a checkstyle issue:
```
[ant:checkstyle] [ERROR] /Users/mickael/github-ws/kafka/clients/src/test/java/org/apache/kafka/common/config/provider/DirectoryConfigProviderTest.java:103:85: ',' is not followed by whitespace. [WhitespaceAfter]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tombentley commented on pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
tombentley commented on pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#issuecomment-670438137
@omkreddy, @mimaison, @rajinisivaram, @kkonstantine, @dajac thanks for voting on the KIP. I'd be grateful for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
mimaison commented on pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#issuecomment-678140872
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tombentley commented on pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
tombentley commented on pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#issuecomment-671424344
@mimaison done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tombentley commented on pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
tombentley commented on pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#issuecomment-678105023
@mimaison fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
mimaison commented on pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#issuecomment-678132390
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
mimaison commented on pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#issuecomment-678131966
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] tombentley commented on a change in pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
tombentley commented on a change in pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#discussion_r468389851
##########
File path: clients/src/main/java/org/apache/kafka/common/config/provider/DirectoryConfigProvider.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config.provider;
+
+import org.apache.kafka.common.config.ConfigData;
+import org.apache.kafka.common.config.ConfigException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.emptyMap;
+
+/**
+ * An implementation of {@link ConfigProvider} based on a directory of files.
+ * Property keys correspond to the names of the regular (i.e. non-directory)
+ * files in a directory given by the path parameter.
+ * Property values are taken from the file contents corresponding to each key.
+ */
+public class DirectoryConfigProvider implements ConfigProvider {
+
+ private static final Logger log = LoggerFactory.getLogger(DirectoryConfigProvider.class);
+
+ @Override
+ public void configure(Map<String, ?> configs) { }
+
+ @Override
+ public void close() throws IOException { }
+
+ /**
+ * Retrieves the data contained in regular files in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path) {
+ return get(path, Files::isRegularFile);
+ }
+
+ /**
+ * Retrieves the data contained in the regular files named by {@code keys} in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @param keys the keys whose values will be retrieved.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path, Set<String> keys) {
+ return get(path, pathname ->
+ Files.isRegularFile(pathname)
+ && keys.contains(pathname.getFileName().toString()));
+ }
+
+ private static ConfigData get(String path, Predicate<Path> fileFilter) {
+ Map<String, String> map = emptyMap();
+ if (path != null && !path.isEmpty()) {
+ Path dir = new File(path).toPath();
+ if (!Files.isDirectory(dir)) {
+ log.warn("The path {} is not a directory", path);
+ } else {
+ try {
+ map = Files.list(dir)
+ .filter(fileFilter)
+ .collect(Collectors.toMap(
+ p -> p.getFileName().toString(),
+ p -> {
+ try {
+ return read(p);
+ } catch (IOException e) {
+ throw new ConfigException("Could not read file " + p + " for property " + p.getFileName(), e);
+ }
Review comment:
Yes, good point. Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
mimaison commented on pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#issuecomment-678651902
Test failures are not related, merging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a change in pull request #9136: KAFKA-10211: Add DirectoryConfigProvider
Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #9136:
URL: https://github.com/apache/kafka/pull/9136#discussion_r468093788
##########
File path: clients/src/main/java/org/apache/kafka/common/config/provider/DirectoryConfigProvider.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.common.config.provider;
+
+import org.apache.kafka.common.config.ConfigData;
+import org.apache.kafka.common.config.ConfigException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import static java.util.Collections.emptyMap;
+
+/**
+ * An implementation of {@link ConfigProvider} based on a directory of files.
+ * Property keys correspond to the names of the regular (i.e. non-directory)
+ * files in a directory given by the path parameter.
+ * Property values are taken from the file contents corresponding to each key.
+ */
+public class DirectoryConfigProvider implements ConfigProvider {
+
+ private static final Logger log = LoggerFactory.getLogger(DirectoryConfigProvider.class);
+
+ @Override
+ public void configure(Map<String, ?> configs) { }
+
+ @Override
+ public void close() throws IOException { }
+
+ /**
+ * Retrieves the data contained in regular files in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path) {
+ return get(path, Files::isRegularFile);
+ }
+
+ /**
+ * Retrieves the data contained in the regular files named by {@code keys} in the directory given by {@code path}.
+ * Non-regular files (such as directories) in the given directory are silently ignored.
+ * @param path the directory where data files reside.
+ * @param keys the keys whose values will be retrieved.
+ * @return the configuration data.
+ */
+ @Override
+ public ConfigData get(String path, Set<String> keys) {
+ return get(path, pathname ->
+ Files.isRegularFile(pathname)
+ && keys.contains(pathname.getFileName().toString()));
+ }
+
+ private static ConfigData get(String path, Predicate<Path> fileFilter) {
+ Map<String, String> map = emptyMap();
+ if (path != null && !path.isEmpty()) {
+ Path dir = new File(path).toPath();
+ if (!Files.isDirectory(dir)) {
+ log.warn("The path {} is not a directory", path);
+ } else {
+ try {
+ map = Files.list(dir)
+ .filter(fileFilter)
+ .collect(Collectors.toMap(
+ p -> p.getFileName().toString(),
+ p -> {
+ try {
+ return read(p);
+ } catch (IOException e) {
+ throw new ConfigException("Could not read file " + p + " for property " + p.getFileName(), e);
+ }
Review comment:
nit: Could we push the `try catch` block into the `read` method? That would streamline the code a little bit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org