You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/04/06 17:57:04 UTC
[GitHub] [iceberg] rizaon opened a new pull request, #4518: core: Provide mechanism to cache manifest file content
rizaon opened a new pull request, #4518:
URL: https://github.com/apache/iceberg/pull/4518
This is a draft PR for ManifestCache implementation.
The `ManifestCache` interface closely follow `com.github.benmanes.caffeine.cache.Cache` interface, with addition of specifying `maxContentLength()`. If stream length is longer than `maxContentLength()`, AvroIO will skip caching it to avoid memory pressure. An example of implementation, `CaffeineManifestCache`, can be seen in TestAvroFileSplit.java.
I will tidy this up and add documentations as we go. Looking forward for any feedback.
Closes #4508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r971815231
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
Review Comment:
Sounds reasonable to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1099703296
[c032b7a](https://github.com/apache/iceberg/pull/4518/commits/c032b7ab2f2db941fe5433c13a38dfcb8bf538ef) implement caching as a new FileIO class, CachingHadoopFileIO. A new Tables class, CachingHadoopTables, is also added to assist with testing.
We tried to avoid `lazyStats()` call as much as possible by checking the cache first before checking for stream length. This comes with a risk of keeping stale data in memory when the actual file is already deleted, as shown in `CachingHiveTableTest.testMissingMetadataWontCauseHang()`. This is probably fine, given the immutable nature of iceberg metadata. Otherwise, application should invalidate cache entry first to force re-read / `lazyStats()` call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1101486057
Hello @rdblue. This PR is ready for review.
Please let me know if there is any new feedback or request. I'm happy to follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r852446403
##########
core/src/main/java/org/apache/iceberg/hadoop/CachingHadoopTables.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hadoop;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.io.FileIO;
+
+public class CachingHadoopTables extends HadoopTables {
Review Comment:
It's a red flag that `HadoopTables` is used. What does caching in `FileIO` have to do with the method used to load tables ? I think those are orthogonal concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] steveloughran commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
steveloughran commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1135593694
> it affects the corresponding time of the first access.
if you are talking about any accessed timestamps in filesystems, know the cloud stores don't have them. and s3 objects creation times are when a multipart upload was initiated, not completed. not relevant for smaller files, but for many mb/gb ones, it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1112641795
@findepi, @alexjo2144, FYI. This could be an option for Trino performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861312858
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
+ }
+
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ close(); // close and discard any previous FileIO instances
+ this.properties = newProperties;
+ initCache();
+ initWrappedIO();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.CONTENT_CACHE_SHARED,
+ CatalogProperties.CONTENT_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getSharedCache(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ public Configuration conf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.hadoopConf = new SerializableConfiguration(conf)::get;
+ }
+
+ @Override
+ public Configuration getConf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void serializeConfWith(Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
+ this.hadoopConf = confSerializer.apply(hadoopConf.get());
+ }
+
+ private void initWrappedIO() {
+ String wrappedIOImpl = PropertyUtil.propertyAsString(properties,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT);
+ this.wrappedIO = CatalogUtil.loadFileIO(wrappedIOImpl, properties, conf());
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, byte[]> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, byte[]>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, byte[]> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private byte[] getFileContent() {
+ try {
+ int totalByteRead = 0;
+ boolean eos = false;
+ int fileLength = (int) getLength();
+ byte[] buf = new byte[fileLength];
+ SeekableInputStream stream = wrappedInputFile.newStream();
+ while (totalByteRead < fileLength && !eos) {
+ int byteRead = stream.read(buf, totalByteRead, fileLength - totalByteRead);
+ if (byteRead < 0) {
+ // Somehow we hit EOS and ended up short. Truncate buf.
+ buf = Arrays.copyOf(buf, totalByteRead);
+ eos = true;
+ } else {
+ totalByteRead += byteRead;
+ }
+ }
+
+ stream.close();
+ return buf;
+ } catch (IOException ex) {
+ throw new RuntimeIOException(ex);
+ }
+ }
+
+ private SeekableInputStream cachedStream() throws IOException {
+ try {
+ byte[] buf = contentCache.cache().get(location(), k -> getFileContent());
+ LOG.debug("Cache stats: {}", contentCache.cache().stats());
+ return new SeekableCachedInputStream(new SeekableByteArrayInputStream(buf));
Review Comment:
Refactored to use `ByteBufferInputStream`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862261481
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
Review Comment:
Style: unnecessary `get` in method name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976131397
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r978814360
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -57,6 +58,68 @@ private CatalogProperties() {}
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * Controls whether to use {@link ContentCache} during manifest file reads or not.
+ *
+ * <p>This value will be ignored and the file cache will be disabled if any of the following is
+ * true:
+ *
+ * <ul>
+ * <li>{@link #IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS} is set to negative number.
+ * <li>{@link #IO_MANIFEST_CACHE_MAX_TOTAL_BYTES} is set to non-positive value.
+ * <li>{@link #IO_MANIFEST_CACHE_MAX_CONTENT_LENGTH} is set to non-positive value.
+ * </ul>
+ */
+ public static final String IO_MANIFEST_CACHE_ENABLED = "io.manifest.cache-enabled";
+
+ public static final boolean IO_MANIFEST_CACHE_ENABLED_DEFAULT = false;
+
+ /**
+ * Controls the duration for which entries in the {@link ContentCache} are cached.
+ *
+ * <p>Behavior of specific values of cache.expiration-interval-ms:
+ *
+ * <ul>
+ * <li>Negative Values - Caching disabled
Review Comment:
nit: I don't think this is true after the changes around enabling/disabling caches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r959442033
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +328,14 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ InputFile inputFile = io.newInputFile(path, length);
Review Comment:
[d86461e](https://github.com/apache/iceberg/pull/4518/commits/d86461ece6643f86a051b4e38fa00d76bcec31d1) address this improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r962267527
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return (buf != null) ? buf.length : wrappedInputFile().getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", location);
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", location);
Review Comment:
Done in [f76fa5d](https://github.com/apache/iceberg/pull/4518/commits/f76fa5d7f8257c63fd90df32fd841ac989cb26af).
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return (buf != null) ? buf.length : wrappedInputFile().getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", location);
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", location);
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
+ try {
+ long fileLength = getLength();
+ long totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile().newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
Review Comment:
Done in [f76fa5d](https://github.com/apache/iceberg/pull/4518/commits/f76fa5d7f8257c63fd90df32fd841ac989cb26af).
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return (buf != null) ? buf.length : wrappedInputFile().getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", location);
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", location);
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
+ try {
+ long fileLength = getLength();
+ long totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile().newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 4MB chunk
+ int bytesToRead = (int) Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // Read less than it should be, possibly hitting EOF.
Review Comment:
[f76fa5d](https://github.com/apache/iceberg/pull/4518/commits/f76fa5d7f8257c63fd90df32fd841ac989cb26af) change this by abandoning content caching if bytesRead is less than bytesToRead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861311809
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
Review Comment:
This remain static in 47b8008. Please let me know if there is a better way to access the cache from Catalog object or outside.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862013772
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 8 * 1024 * 1024; // 8MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffer;
+
+ private CacheEntry(int length, List<ByteBuffer> buffer) {
+ this.length = length;
+ this.buffer = buffer;
+ }
+
+ public int length() {
+ return length;
+ }
+
+ public List<ByteBuffer> buffer() {
+ return buffer;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length())
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length() : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
+ try {
+ int fileLength = (int) getLength();
+ int totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile.newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 8MB chunk
+ int bytesToRead = Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // We are reading less than we should, possibly hitting EOF.
+ // Truncate buf and break from the loop.
+ buf = Arrays.copyOf(buf, bytesRead);
Review Comment:
I'm keeping the copying method in [1b98dc2](https://github.com/apache/iceberg/pull/4518/commits/1b98dc218a0e7b2ab66dead1aa2a40b623aaba3e), but reduce the chunk size from 8MB to 4MB.
The intention is that, if we indeed unexpectedly hit EOF early, the worst case is we are doing slightly under 4MB copy rather than slightly under 8MB copy. But if `getLength` is accurate, we should not reach this copying code.
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 8 * 1024 * 1024; // 8MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffer;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r852990871
##########
core/src/main/java/org/apache/iceberg/hadoop/CachingHadoopFileIO.java:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hadoop;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Variant of {@link HadoopFileIO} that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingHadoopFileIO extends HadoopFileIO {
Review Comment:
Apologies, you're right. This should be generic. I fix this by making it wrap another FileIO in [8cf5196](https://github.com/apache/iceberg/pull/4518/commits/8cf5196b80bbd0bbd41db23dd072df2533c4be71).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r852991634
##########
core/src/main/java/org/apache/iceberg/hadoop/CachingHadoopTables.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hadoop;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.io.FileIO;
+
+public class CachingHadoopTables extends HadoopTables {
Review Comment:
This is now dropped. We are now testing directly against `CachingFileIO`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r865450006
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
Review Comment:
Renamed to `cacheEntry()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r978820967
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +339,58 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ boolean enabled = cachingEnabled(io);
+
+ if (enabled) {
+ ContentCache cache = contentCache(io);
+ Preconditions.checkNotNull(cache);
+ LOG.debug("FileIO-level cache stats: {}", CONTENT_CACHES.stats());
+ return cache.tryCache(io, path, length);
+ }
+
+ // caching is not enable for this io or caught RuntimeException.
+ return io.newInputFile(path, length);
+ }
+
+ private static int maxFileIO() {
+ String value = System.getProperty(SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO);
+ if (value != null) {
+ try {
+ return Integer.parseUnsignedInt(value);
+ } catch (NumberFormatException e) {
+ // will return the default
+ }
+ }
+ return SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO_DEFAULT;
+ }
+
+ public static boolean cachingEnabled(FileIO io) {
Review Comment:
This can be package level protection.
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +339,58 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ boolean enabled = cachingEnabled(io);
+
+ if (enabled) {
+ ContentCache cache = contentCache(io);
+ Preconditions.checkNotNull(cache);
+ LOG.debug("FileIO-level cache stats: {}", CONTENT_CACHES.stats());
+ return cache.tryCache(io, path, length);
+ }
+
+ // caching is not enable for this io or caught RuntimeException.
+ return io.newInputFile(path, length);
+ }
+
+ private static int maxFileIO() {
+ String value = System.getProperty(SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO);
+ if (value != null) {
+ try {
+ return Integer.parseUnsignedInt(value);
+ } catch (NumberFormatException e) {
+ // will return the default
+ }
+ }
+ return SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO_DEFAULT;
+ }
+
+ public static boolean cachingEnabled(FileIO io) {
+ return PropertyUtil.propertyAsBoolean(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED,
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED_DEFAULT);
+ }
+
+ public static long cacheDurationMs(FileIO io) {
Review Comment:
This can be package level protection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r978821360
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +339,58 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ boolean enabled = cachingEnabled(io);
+
+ if (enabled) {
+ ContentCache cache = contentCache(io);
+ Preconditions.checkNotNull(cache);
+ LOG.debug("FileIO-level cache stats: {}", CONTENT_CACHES.stats());
+ return cache.tryCache(io, path, length);
+ }
+
+ // caching is not enable for this io or caught RuntimeException.
+ return io.newInputFile(path, length);
+ }
+
+ private static int maxFileIO() {
+ String value = System.getProperty(SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO);
+ if (value != null) {
+ try {
+ return Integer.parseUnsignedInt(value);
+ } catch (NumberFormatException e) {
+ // will return the default
+ }
+ }
+ return SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO_DEFAULT;
+ }
+
+ public static boolean cachingEnabled(FileIO io) {
+ return PropertyUtil.propertyAsBoolean(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED,
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED_DEFAULT);
+ }
+
+ public static long cacheDurationMs(FileIO io) {
+ return PropertyUtil.propertyAsLong(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ }
+
+ public static long cacheTotalBytes(FileIO io) {
+ return PropertyUtil.propertyAsLong(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_MANIFEST_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ }
+
+ public static long cacheMaxContentLength(FileIO io) {
Review Comment:
This can be package level protection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1177100084
Hi @danielcweeks , thank you for the update. I will check PR #5207 and study how we can utilize it.
So with that PR, am I correct that we should implement the caching in ManifestFiles.java instead of creating a new CachingFileIO.java as this PR did?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976148368
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ durationMs);
+ } else if (totalBytes <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ totalBytes);
+ } else if (contentLength <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ contentLength);
+ } else {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ }
+
+ return null;
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("expireAfterAccessMs", expireAfterAccessMs)
+ .add("maxContentLength", maxContentLength)
+ .add("maxTotalBytes", maxTotalBytes)
+ .add("cacheStats", cache.stats())
+ .toString();
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ if (buf != null) {
+ return buf.length;
+ } else if (fallbackInputFile != null) {
+ return fallbackInputFile.getLength();
+ } else {
+ return length;
+ }
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ } catch (IOException e) {
+ throw new RuntimeIOException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
Review Comment:
Filed #5814 as a follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1109315236
Got it, thank you. Will rebase and update this PR once they are merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861310326
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -53,6 +56,57 @@ private CatalogProperties() {
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * When using {@link CachingFileIO} as io-impl, define the internal FileIO that is
+ * wrapped by CachingFileIO.
+ * <p>
+ * Default to {@link HadoopFileIO} if not defined.
+ */
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL = "cache.content.wrapped-io-impl";
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT = HadoopFileIO.class.getName();
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861317041
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 8 * 1024 * 1024; // 8MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffer;
+
+ private CacheEntry(int length, List<ByteBuffer> buffer) {
+ this.length = length;
+ this.buffer = buffer;
+ }
+
+ public int length() {
+ return length;
+ }
+
+ public List<ByteBuffer> buffer() {
+ return buffer;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length())
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length() : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
+ try {
+ int fileLength = (int) getLength();
+ int totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile.newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 8MB chunk
+ int bytesToRead = Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // We are reading less than we should, possibly hitting EOF.
+ // Truncate buf and break from the loop.
+ buf = Arrays.copyOf(buf, bytesRead);
Review Comment:
Rather than copying, can't this just create the buffer with a smaller limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861312187
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
Review Comment:
Renamed to `getOrCreate`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861331100
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 8 * 1024 * 1024; // 8MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffer;
+
+ private CacheEntry(int length, List<ByteBuffer> buffer) {
+ this.length = length;
+ this.buffer = buffer;
+ }
+
+ public int length() {
+ return length;
+ }
+
+ public List<ByteBuffer> buffer() {
+ return buffer;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length())
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length() : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
+ try {
+ int fileLength = (int) getLength();
+ int totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile.newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 8MB chunk
+ int bytesToRead = Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // We are reading less than we should, possibly hitting EOF.
+ // Truncate buf and break from the loop.
+ buf = Arrays.copyOf(buf, bytesRead);
Review Comment:
Do you mean to use second variant of `ByteBuffer.wrap`? Like `ByteBuffer.wrap(buf, 0, bytesRead)`?
I suppose we can, but that does not seem to reduce the backing array size?
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 8 * 1024 * 1024; // 8MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffer;
Review Comment:
Yes, `CacheEntry` can be private.
I just wish that `ContentCache` itself is accessible to the outside world so that query engine can force cache invalidation by themself if necessary (ie., upon [REFRESH](https://impala.apache.org/docs/build3x/html/topics/impala_refresh.html) query).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1092128396
[485e2ab](https://github.com/apache/iceberg/pull/4518/commits/485e2abf4bb3c67f7ab0db2e5cf2e78397bf25db) Switch ManifestCache class to use Guava Cache instead. This is easier to do in iceberg-api since we can simply include the Guava cache classes into the shadow jar of iceberg-bundled-guava.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862260352
##########
core/src/main/java/org/apache/iceberg/CatalogUtil.java:
##########
@@ -272,6 +273,13 @@ public static FileIO loadFileIO(
configureHadoopConf(fileIO, hadoopConf);
fileIO.initialize(properties);
+
+ // Check if we should wrap this with CachingFileIO
Review Comment:
Minor: personal pronouns like "I" and "we" usually make comments longer and harder to read. More direct statements like `Check whether to wrap with CachingFileIO` are preferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855664798
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
Review Comment:
Is there a way to pass in the cache rather than making it static?
@jackye1995, do you have any ideas here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r856686277
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
+ }
+
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ close(); // close and discard any previous FileIO instances
+ this.properties = newProperties;
+ initCache();
+ initWrappedIO();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.CONTENT_CACHE_SHARED,
+ CatalogProperties.CONTENT_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getSharedCache(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ public Configuration conf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.hadoopConf = new SerializableConfiguration(conf)::get;
+ }
+
+ @Override
+ public Configuration getConf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void serializeConfWith(Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
+ this.hadoopConf = confSerializer.apply(hadoopConf.get());
+ }
+
+ private void initWrappedIO() {
+ String wrappedIOImpl = PropertyUtil.propertyAsString(properties,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT);
+ this.wrappedIO = CatalogUtil.loadFileIO(wrappedIOImpl, properties, conf());
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, byte[]> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, byte[]>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, byte[]> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private byte[] getFileContent() {
+ try {
+ int totalByteRead = 0;
+ boolean eos = false;
+ int fileLength = (int) getLength();
+ byte[] buf = new byte[fileLength];
Review Comment:
Are you referring to this class?
https://github.com/apache/parquet-mr/blob/master/parquet-common/src/main/java/org/apache/parquet/bytes/ByteBufferInputStream.java
I look around, but can't seem to find class that can adapt that `ByteBufferInputStream` to a `SeekableInputStream` needed for `newStream()` later. Can you confirm that I should copy that class (possibly with `SingleBufferInputStream` and `MultiBufferInputStream` as well), and implement `seek()` on it so it can be adapted to `SeekableInputStream`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855665352
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
Review Comment:
Why does this reference a Hadoop Configuration? It seems like this doesn't need to rely on Hadoop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1094370689
Switching to use more Guava classes is probably not a good idea.
@rizaon, do you have use cases where this has helped? If so, what were they? This adds quite a bit of complexity and memory overhead that we purposely avoided up to now so I want to make sure it is worth the change.
Have you considered adding a caching FileIO instance? That could be used for more use cases than just manifest files and job planning. For example, a FileIO read-through cache could cache delete files that might be reused for tasks. This could be configured by file path, making it easy to determine what to cache. Plus, we could use more options than just in-memory, like a distributed cache or local disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861314480
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
+ }
+
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ close(); // close and discard any previous FileIO instances
+ this.properties = newProperties;
+ initCache();
+ initWrappedIO();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.CONTENT_CACHE_SHARED,
+ CatalogProperties.CONTENT_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getSharedCache(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ public Configuration conf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.hadoopConf = new SerializableConfiguration(conf)::get;
+ }
+
+ @Override
+ public Configuration getConf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void serializeConfWith(Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
+ this.hadoopConf = confSerializer.apply(hadoopConf.get());
+ }
+
+ private void initWrappedIO() {
+ String wrappedIOImpl = PropertyUtil.propertyAsString(properties,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT);
+ this.wrappedIO = CatalogUtil.loadFileIO(wrappedIOImpl, properties, conf());
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, byte[]> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, byte[]>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, byte[]> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private byte[] getFileContent() {
+ try {
+ int totalByteRead = 0;
+ boolean eos = false;
+ int fileLength = (int) getLength();
+ byte[] buf = new byte[fileLength];
Review Comment:
Refactored to use `IOUtil.readRemaining`. However, it still relies on `getLength` being accurate. Please let me know if it should keep looping until EOF met.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861317973
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 8 * 1024 * 1024; // 8MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffer;
Review Comment:
I think this should be `buffers` since it is a list.
Also, if this class is private, then it's okay to access fields directly. But if it is public then everything should use getter methods. I don't see a reason for the class to be public, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r962269646
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -43,6 +47,30 @@ private ManifestFiles() {}
GenericManifestFile.class.getName(),
ManifestFile.PARTITION_SUMMARY_TYPE,
GenericPartitionFieldSummary.class.getName()));
+ private static final int CONTENT_CACHES_EXPIRATION_INTERVAL_SECOND = 60 * 60; // 1 hour
Review Comment:
[f76fa5d](https://github.com/apache/iceberg/pull/4518/commits/f76fa5d7f8257c63fd90df32fd841ac989cb26af) change `CONTENT_CACHE` to evict by size and reference.
Since content caching is per catalog/FileIO, different FileIO might have different caching configuration, including expiration time. This expiration time can be longer than 60s. Thus, using time-based eviction for `CONTENT_CACHE` might conflict with with per-FileIO cache configuration. I don't think we can use maximum weight as well for `CONTENT_CACHE` since according to the [documentation](https://github.com/ben-manes/caffeine/wiki/Eviction) entry's weight is only calculated on entry creation and update time, and static thereafter. All `CONTENT_CACHE` entries then will have 0 weight initially and potentially stay that way.
[f76fa5d](https://github.com/apache/iceberg/pull/4518/commits/f76fa5d7f8257c63fd90df32fd841ac989cb26af) add default upper bound to 8 FileIO to cache (no specific reason about why it should be 8, I'm open to change it). Additionally, it also use `weakKeys()` to allow garbage collection of an entry if there are no other strong references to the keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1177773675
@rizaon because we want to isolate the caching to metadata files only, I think MetadataFiles.java would be a good place to start. We might want some utilities so if we find we want to cache FileIO elsewhere, we can reuse the same approach, but this seems like a full FileIO implementation isn't necessary (and introduces other problems).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862260651
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
Review Comment:
This should be `estimatedCacheSize`, right? This isn't caching the size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r978821160
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +339,58 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ boolean enabled = cachingEnabled(io);
+
+ if (enabled) {
+ ContentCache cache = contentCache(io);
+ Preconditions.checkNotNull(cache);
+ LOG.debug("FileIO-level cache stats: {}", CONTENT_CACHES.stats());
+ return cache.tryCache(io, path, length);
+ }
+
+ // caching is not enable for this io or caught RuntimeException.
+ return io.newInputFile(path, length);
+ }
+
+ private static int maxFileIO() {
+ String value = System.getProperty(SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO);
+ if (value != null) {
+ try {
+ return Integer.parseUnsignedInt(value);
+ } catch (NumberFormatException e) {
+ // will return the default
+ }
+ }
+ return SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO_DEFAULT;
+ }
+
+ public static boolean cachingEnabled(FileIO io) {
+ return PropertyUtil.propertyAsBoolean(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED,
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED_DEFAULT);
+ }
+
+ public static long cacheDurationMs(FileIO io) {
+ return PropertyUtil.propertyAsLong(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ }
+
+ public static long cacheTotalBytes(FileIO io) {
Review Comment:
This can be package level protection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r978814840
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -57,6 +58,68 @@ private CatalogProperties() {}
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * Controls whether to use {@link ContentCache} during manifest file reads or not.
+ *
+ * <p>This value will be ignored and the file cache will be disabled if any of the following is
+ * true:
+ *
+ * <ul>
+ * <li>{@link #IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS} is set to negative number.
+ * <li>{@link #IO_MANIFEST_CACHE_MAX_TOTAL_BYTES} is set to non-positive value.
+ * <li>{@link #IO_MANIFEST_CACHE_MAX_CONTENT_LENGTH} is set to non-positive value.
+ * </ul>
+ */
+ public static final String IO_MANIFEST_CACHE_ENABLED = "io.manifest.cache-enabled";
+
+ public static final boolean IO_MANIFEST_CACHE_ENABLED_DEFAULT = false;
+
+ /**
+ * Controls the duration for which entries in the {@link ContentCache} are cached.
+ *
+ * <p>Behavior of specific values of cache.expiration-interval-ms:
+ *
+ * <ul>
+ * <li>Negative Values - Caching disabled
+ * <li>Zero - Cache entries expires only if it gets evicted due to memory pressure from {@link
+ * #IO_MANIFEST_CACHE_MAX_TOTAL_BYTES} setting.
+ * <li>Positive Values - Cache entries expire if not accessed via the cache after this many
+ * milliseconds
+ * </ul>
+ */
+ public static final String IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS =
+ "io.manifest.cache.expiration-interval-ms";
+
+ public static final long IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT =
+ TimeUnit.SECONDS.toMillis(60);
+
+ /**
+ * Controls the maximum total amount of bytes to cache in {@link ContentCache}.
+ *
+ * <p>Behavior of specific values of cache.content.max-total-bytes:
+ *
+ * <ul>
+ * <li>Non-Positive Values - Caching disabled
Review Comment:
This also may be inconsistent with the behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855674533
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
+ }
+
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ close(); // close and discard any previous FileIO instances
+ this.properties = newProperties;
+ initCache();
+ initWrappedIO();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.CONTENT_CACHE_SHARED,
+ CatalogProperties.CONTENT_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getSharedCache(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ public Configuration conf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.hadoopConf = new SerializableConfiguration(conf)::get;
+ }
+
+ @Override
+ public Configuration getConf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void serializeConfWith(Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
+ this.hadoopConf = confSerializer.apply(hadoopConf.get());
+ }
+
+ private void initWrappedIO() {
+ String wrappedIOImpl = PropertyUtil.propertyAsString(properties,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT);
+ this.wrappedIO = CatalogUtil.loadFileIO(wrappedIOImpl, properties, conf());
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, byte[]> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, byte[]>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, byte[]> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private byte[] getFileContent() {
+ try {
+ int totalByteRead = 0;
+ boolean eos = false;
+ int fileLength = (int) getLength();
+ byte[] buf = new byte[fileLength];
Review Comment:
Yeah, thinking about this more, I think it should allocate small buffers. In a loop, allocate a buffer that is 4 or 8 MB, then call `ByteStreams.readFully` to fill the buffer, and then wrap it in a `ByteBuffer`. Then use `ByteBufferInputStream` and the associated classes from Parquet (probably just copy those into `util` along with tests since we will use them in other places).
You may need a cache entry to have a fast size lookup for the caffeine `Cache`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855670659
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
+ }
+
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ close(); // close and discard any previous FileIO instances
+ this.properties = newProperties;
+ initCache();
+ initWrappedIO();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.CONTENT_CACHE_SHARED,
+ CatalogProperties.CONTENT_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getSharedCache(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ public Configuration conf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.hadoopConf = new SerializableConfiguration(conf)::get;
+ }
+
+ @Override
+ public Configuration getConf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void serializeConfWith(Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
+ this.hadoopConf = confSerializer.apply(hadoopConf.get());
+ }
+
+ private void initWrappedIO() {
+ String wrappedIOImpl = PropertyUtil.propertyAsString(properties,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT);
+ this.wrappedIO = CatalogUtil.loadFileIO(wrappedIOImpl, properties, conf());
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, byte[]> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, byte[]>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, byte[]> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private byte[] getFileContent() {
+ try {
+ int totalByteRead = 0;
+ boolean eos = false;
+ int fileLength = (int) getLength();
+ byte[] buf = new byte[fileLength];
+ SeekableInputStream stream = wrappedInputFile.newStream();
+ while (totalByteRead < fileLength && !eos) {
Review Comment:
Can this use `ByteStreams.readFully` instead of implementing its own `readFully` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861314579
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
+ }
+
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ close(); // close and discard any previous FileIO instances
+ this.properties = newProperties;
+ initCache();
+ initWrappedIO();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.CONTENT_CACHE_SHARED,
+ CatalogProperties.CONTENT_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getSharedCache(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ public Configuration conf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.hadoopConf = new SerializableConfiguration(conf)::get;
+ }
+
+ @Override
+ public Configuration getConf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void serializeConfWith(Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
+ this.hadoopConf = confSerializer.apply(hadoopConf.get());
+ }
+
+ private void initWrappedIO() {
+ String wrappedIOImpl = PropertyUtil.propertyAsString(properties,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT);
+ this.wrappedIO = CatalogUtil.loadFileIO(wrappedIOImpl, properties, conf());
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, byte[]> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, byte[]>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, byte[]> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1201545507
337bfe8 is a rebase + `spotlessApply` of 4d4c4fa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r973750356
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -57,6 +58,64 @@ private CatalogProperties() {}
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * Controls whether to use {@link ContentCache} during manifest file reads or not.
+ *
+ * <p>This value will be ignored and the file cache will be disabled if any of the following is
+ * true:
+ *
+ * <ul>
+ * <li>{@link #IO_CACHE_EXPIRATION_INTERVAL_MS} is set to negative number.
+ * <li>{@link #IO_CACHE_MAX_TOTAL_BYTES} is set to non-positive value.
+ * <li>{@link #IO_CACHE_MAX_CONTENT_LENGTH} is set to non-positive value.
+ * </ul>
+ */
+ public static final String IO_CACHE_ENABLED = "io.cache-enabled";
Review Comment:
I feel like we need to update these property names as it's not clear what is being cached. I would suggest `io.manifest.cache-enabled`
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
Review Comment:
These three values should probably be a validation check as opposed to equivalent to disabling the cache. (See `ValidationException.check` or `Preconditions`). If caching is enabled, these should all be set to valid values.
##########
core/src/main/java/org/apache/iceberg/SystemProperties.java:
##########
@@ -33,6 +33,14 @@ private SystemProperties() {}
/** Whether to use the shared worker pool when planning table scans. */
public static final String SCAN_THREAD_POOL_ENABLED = "iceberg.scan.plan-in-worker-pool";
+ /**
+ * Maximum number of distinct {@link org.apache.iceberg.io.FileIO} that is allowed to have
+ * associated {@link org.apache.iceberg.io.ContentCache} in memory at a time.
+ */
+ public static final String IO_CACHE_MAX_FILEIO = "iceberg.io.cache.max-fileio";
Review Comment:
Similar to the other properties, this is manifest specific so we probably want to chance this to `iceberg.io.manifest.cache.fileio-max`
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
Review Comment:
ContentCache is rather generic and I feel like we should have some javadoc explaining what this cache is and how it is used.
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ durationMs);
+ } else if (totalBytes <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ totalBytes);
+ } else if (contentLength <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ contentLength);
+ } else {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ }
+
+ return null;
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("expireAfterAccessMs", expireAfterAccessMs)
+ .add("maxContentLength", maxContentLength)
+ .add("maxTotalBytes", maxTotalBytes)
+ .add("cacheStats", cache.stats())
+ .toString();
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ if (buf != null) {
+ return buf.length;
+ } else if (fallbackInputFile != null) {
+ return fallbackInputFile.getLength();
+ } else {
+ return length;
+ }
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ } catch (IOException e) {
+ throw new RuntimeIOException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
+ long start = System.currentTimeMillis();
+ try (SeekableInputStream stream = wrappedInputFile().newStream()) {
+ long fileLength = getLength();
+ long totalBytesToRead = fileLength;
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 4MB chunk
+ int bytesToRead = (int) Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // Read less than it should be, possibly hitting EOF. Abandon caching by throwing
+ // IOException and let the caller fallback to non-caching input file.
+ throw new IOException(
+ "Expected to read "
+ + fileLength
+ + " bytes, but only "
+ + (fileLength - totalBytesToRead)
+ + " bytes "
+ + "read.");
+ } else {
+ buffers.add(ByteBuffer.wrap(buf));
+ }
+ }
+
+ CacheEntry newEntry = new CacheEntry(fileLength - totalBytesToRead, buffers);
+ LOG.debug("cacheEntry took {} ms for {}", (System.currentTimeMillis() - start), location);
+ return newEntry;
+ } catch (IOException ex) {
+ throw new RuntimeIOException(ex);
Review Comment:
(Deprecation) switch to `UncheckedIOException`
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -44,6 +51,28 @@ private ManifestFiles() {}
ManifestFile.PARTITION_SUMMARY_TYPE,
GenericPartitionFieldSummary.class.getName()));
+ private static final Cache<FileIO, ContentCache> CONTENT_CACHES =
+ Caffeine.newBuilder().weakKeys().maximumSize(maxFileIO()).recordStats().build();
Review Comment:
If we're worried about reclaiming memory under pressure, don't we need `weakValues()` as that's where the content is stored?
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -44,6 +51,28 @@ private ManifestFiles() {}
ManifestFile.PARTITION_SUMMARY_TYPE,
GenericPartitionFieldSummary.class.getName()));
+ private static final Cache<FileIO, ContentCache> CONTENT_CACHES =
+ Caffeine.newBuilder().weakKeys().maximumSize(maxFileIO()).recordStats().build();
+
+ /**
+ * Get or create a manifest file cache for a given FileIO.
+ *
+ * <p>Returned cache object will be kept in memory until {@link #dropCache(FileIO)} is called.
+ * Return null if the given FileIO properties does not allow caching.
+ *
+ * @param fileIO A FileIO.
+ * @return A manifest file cache or null if the given FileIO properties does not allow caching.
+ */
+ public static ContentCache getOrCreateCache(FileIO fileIO) {
Review Comment:
After moving the `enable` check, I don't think we'll need this try/catch
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ durationMs);
+ } else if (totalBytes <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ totalBytes);
+ } else if (contentLength <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ contentLength);
+ } else {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ }
+
+ return null;
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("expireAfterAccessMs", expireAfterAccessMs)
+ .add("maxContentLength", maxContentLength)
+ .add("maxTotalBytes", maxTotalBytes)
+ .add("cacheStats", cache.stats())
+ .toString();
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ if (buf != null) {
+ return buf.length;
+ } else if (fallbackInputFile != null) {
+ return fallbackInputFile.getLength();
+ } else {
+ return length;
+ }
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ } catch (IOException e) {
+ throw new RuntimeIOException(
Review Comment:
deprecation: use UncheckedIOException
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -57,6 +58,64 @@ private CatalogProperties() {}
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * Controls whether to use {@link ContentCache} during manifest file reads or not.
+ *
+ * <p>This value will be ignored and the file cache will be disabled if any of the following is
+ * true:
+ *
+ * <ul>
+ * <li>{@link #IO_CACHE_EXPIRATION_INTERVAL_MS} is set to negative number.
+ * <li>{@link #IO_CACHE_MAX_TOTAL_BYTES} is set to non-positive value.
+ * <li>{@link #IO_CACHE_MAX_CONTENT_LENGTH} is set to non-positive value.
+ * </ul>
+ */
+ public static final String IO_CACHE_ENABLED = "io.cache-enabled";
Review Comment:
Same for all the other properties.
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
Review Comment:
As mentioned in the comment on ManifestFiles, I would move this `enabled` check out and make the remainder just validation checks.
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +329,30 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ try {
+ ContentCache cache = getOrCreateCache(io);
Review Comment:
I would move the `enabled` check here so we're not calling into the ContentCache and using `null` to the main code path behavior. This would also get rid of the `getOrCreateCache` method above.
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ durationMs);
+ } else if (totalBytes <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ totalBytes);
+ } else if (contentLength <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ contentLength);
+ } else {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ }
+
+ return null;
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("expireAfterAccessMs", expireAfterAccessMs)
+ .add("maxContentLength", maxContentLength)
+ .add("maxTotalBytes", maxTotalBytes)
+ .add("cacheStats", cache.stats())
+ .toString();
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ if (buf != null) {
+ return buf.length;
+ } else if (fallbackInputFile != null) {
+ return fallbackInputFile.getLength();
+ } else {
+ return length;
+ }
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ } catch (IOException e) {
+ throw new RuntimeIOException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
+ long start = System.currentTimeMillis();
+ try (SeekableInputStream stream = wrappedInputFile().newStream()) {
+ long fileLength = getLength();
+ long totalBytesToRead = fileLength;
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 4MB chunk
+ int bytesToRead = (int) Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // Read less than it should be, possibly hitting EOF. Abandon caching by throwing
+ // IOException and let the caller fallback to non-caching input file.
+ throw new IOException(
+ "Expected to read "
+ + fileLength
+ + " bytes, but only "
+ + (fileLength - totalBytesToRead)
+ + " bytes "
+ + "read.");
+ } else {
+ buffers.add(ByteBuffer.wrap(buf));
+ }
+ }
+
+ CacheEntry newEntry = new CacheEntry(fileLength - totalBytesToRead, buffers);
Review Comment:
`totalBytesToRead` should always be zero so this should just be `fileLength`
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ durationMs);
+ } else if (totalBytes <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ totalBytes);
+ } else if (contentLength <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ contentLength);
+ } else {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ }
+
+ return null;
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("expireAfterAccessMs", expireAfterAccessMs)
+ .add("maxContentLength", maxContentLength)
+ .add("maxTotalBytes", maxTotalBytes)
+ .add("cacheStats", cache.stats())
+ .toString();
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ if (buf != null) {
+ return buf.length;
+ } else if (fallbackInputFile != null) {
+ return fallbackInputFile.getLength();
+ } else {
+ return length;
+ }
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ } catch (IOException e) {
+ throw new RuntimeIOException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
Review Comment:
Something to think about (possibly create an issue), but this isn't really read-through, but rather read-ahead since we're reading the whole file before allowing the reader to process the manifest file. We could improve this by making it a read-through buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r978887511
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -57,6 +58,68 @@ private CatalogProperties() {}
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * Controls whether to use {@link ContentCache} during manifest file reads or not.
+ *
+ * <p>This value will be ignored and the file cache will be disabled if any of the following is
+ * true:
+ *
+ * <ul>
+ * <li>{@link #IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS} is set to negative number.
+ * <li>{@link #IO_MANIFEST_CACHE_MAX_TOTAL_BYTES} is set to non-positive value.
+ * <li>{@link #IO_MANIFEST_CACHE_MAX_CONTENT_LENGTH} is set to non-positive value.
+ * </ul>
+ */
+ public static final String IO_MANIFEST_CACHE_ENABLED = "io.manifest.cache-enabled";
+
+ public static final boolean IO_MANIFEST_CACHE_ENABLED_DEFAULT = false;
+
+ /**
+ * Controls the duration for which entries in the {@link ContentCache} are cached.
+ *
+ * <p>Behavior of specific values of cache.expiration-interval-ms:
+ *
+ * <ul>
+ * <li>Negative Values - Caching disabled
Review Comment:
Clarified the documentation.
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -57,6 +58,68 @@ private CatalogProperties() {}
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * Controls whether to use {@link ContentCache} during manifest file reads or not.
+ *
+ * <p>This value will be ignored and the file cache will be disabled if any of the following is
+ * true:
+ *
+ * <ul>
+ * <li>{@link #IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS} is set to negative number.
+ * <li>{@link #IO_MANIFEST_CACHE_MAX_TOTAL_BYTES} is set to non-positive value.
+ * <li>{@link #IO_MANIFEST_CACHE_MAX_CONTENT_LENGTH} is set to non-positive value.
+ * </ul>
+ */
+ public static final String IO_MANIFEST_CACHE_ENABLED = "io.manifest.cache-enabled";
+
+ public static final boolean IO_MANIFEST_CACHE_ENABLED_DEFAULT = false;
+
+ /**
+ * Controls the duration for which entries in the {@link ContentCache} are cached.
+ *
+ * <p>Behavior of specific values of cache.expiration-interval-ms:
+ *
+ * <ul>
+ * <li>Negative Values - Caching disabled
+ * <li>Zero - Cache entries expires only if it gets evicted due to memory pressure from {@link
+ * #IO_MANIFEST_CACHE_MAX_TOTAL_BYTES} setting.
+ * <li>Positive Values - Cache entries expire if not accessed via the cache after this many
+ * milliseconds
+ * </ul>
+ */
+ public static final String IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS =
+ "io.manifest.cache.expiration-interval-ms";
+
+ public static final long IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT =
+ TimeUnit.SECONDS.toMillis(60);
+
+ /**
+ * Controls the maximum total amount of bytes to cache in {@link ContentCache}.
+ *
+ * <p>Behavior of specific values of cache.content.max-total-bytes:
+ *
+ * <ul>
+ * <li>Non-Positive Values - Caching disabled
Review Comment:
Clarified the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r865446956
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
Review Comment:
Done
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r865447311
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
Review Comment:
Added more API method to avoid accessing internal `cache`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r865450219
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
+ try {
+ int fileLength = (int) getLength();
Review Comment:
Changed to `long`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1095389667
Hi @rdblue, thank you for your feedback.
We found a slow query compilation issue against the Iceberg table in our recent Apache Impala build. Impala uses Iceberg's HiveCatalog and HadoopFileIO instance with an S3A input stream to access data from S3. We did a full 10 TB TPC-DS benchmark and found that query compilation can go for several seconds, while it used to be less than a second with native hive tables. This slowness in single query compilation is due to the requirement to call planFiles several times, even for scan nodes targetting the same table. We also see several socket read operations that spend hundreds of milliseconds during planFiles, presumably due to S3A HTTP HEAD request overhead and backward seek overhead (issue #4508). This is especially hurt fast-running queries.
We tried this caching solution and it help speed up Impala query compilation almost 5x faster compared to without on Iceberg tables. Our original solution, however, is to put a Caffeine cache as a singleton in AvroIO.java. I thought it is better to supply the cache from outside.
I have not considered the solution of adding a caching FileIO instance. I'm pretty new to the Iceberg codebase but interested to follow up on that if it can yield a better integration. Will it require a new class of Catalog/Table as well, or can we improve on the existing HiveCatalog & HadoopFileIO?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1095393793
A relevant Impala's JIRA is here:
https://issues.apache.org/jira/browse/IMPALA-11171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976128390
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -57,6 +58,64 @@ private CatalogProperties() {}
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * Controls whether to use {@link ContentCache} during manifest file reads or not.
+ *
+ * <p>This value will be ignored and the file cache will be disabled if any of the following is
+ * true:
+ *
+ * <ul>
+ * <li>{@link #IO_CACHE_EXPIRATION_INTERVAL_MS} is set to negative number.
+ * <li>{@link #IO_CACHE_MAX_TOTAL_BYTES} is set to non-positive value.
+ * <li>{@link #IO_CACHE_MAX_CONTENT_LENGTH} is set to non-positive value.
+ * </ul>
+ */
+ public static final String IO_CACHE_ENABLED = "io.cache-enabled";
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1237636661
Core test failed at `HadoopFileIOTest > testHadoopFileIOKryoSerialization()`. Should I do rebase to pick up the fix from #5437 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] steveloughran commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
steveloughran commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r864026701
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
+ try {
+ int fileLength = (int) getLength();
Review Comment:
something bigger than an int long should automatically be excluded from caching, presumably. though if someone did want to declare a cache of > 2GB then it may be needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r863388111
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
Review Comment:
Hi @danielcweeks , thank you for your feedback!
I can't seem to find an easy way to pass catalog properties (where we define caching related knobs) to ManifestFiles.java. How about we let the CachingFileIO wrapping just like now, but create new method, say `newMetadataInputFile()`, that specially used by `ManifestReader.read()`?
Maybe modification like this in ManifestFiles.java:85
```
InputFile file = (io instanceof CachingFileIO)
? ((CachingFileIO) io).newMetadataInputFile(manifest.path())
: io.newInputFile(manifest.path());
```
For the `newInputFile()` implementation by CachingFileIO, we can just pass-through it to the inner FileIO. Thus, it ensure that caching only applies to metadata files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r865450793
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
+ try {
+ int fileLength = (int) getLength();
+ int totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile.newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 4MB chunk
+ int bytesToRead = Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // We are reading less than we should, possibly hitting EOF.
+ // Truncate buffer and break from the loop. In worst case, this will do byte copying of slightly under 4MB.
+ byte[] smallerBuf = new byte[bytesRead];
+ System.arraycopy(buf, 0, smallerBuf, 0, bytesRead);
Review Comment:
Updated by wrapping it with smaller ByteBuffer limit instead of truncating the array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r865449819
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
Review Comment:
Removed `getLengthChecked()`. However, without checking the cause, `newStream()` will throw `RuntimeIOException` instead of `NotFoundException`.
Updated assertion at `CachingHiveTableTest.testMissingMetadataWontCauseHang` accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r865448620
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
Review Comment:
After looking around, I ended up go the regex route.
There seems to be many call sites where Iceberg is reading metadata files, not just in ManifestFiles.java. Among them are:
```
$ git grep -e "newInputFile.*manifest" -e "newInputFile.*meta"
core/src/main/java/org/apache/iceberg/AllManifestsTable.java: .withInputFile(io.newInputFile(snap.manifestListLocation()))
core/src/main/java/org/apache/iceberg/AllManifestsTable.java: io.newInputFile(tableOps().current().metadataFileLocation()),
core/src/main/java/org/apache/iceberg/AllManifestsTable.java: .read(io.newInputFile(manifestListTask.file().path().toString()))
core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java: InputFile toCopy = ops.io().newInputFile(manifest.path());
core/src/main/java/org/apache/iceberg/BaseSnapshot.java: this.allManifests = ManifestLists.read(io.newInputFile(manifestListLocation));
core/src/main/java/org/apache/iceberg/FastAppend.java: InputFile toCopy = ops.io().newInputFile(manifest.path());
core/src/main/java/org/apache/iceberg/HistoryTable.java: ops.io().newInputFile(ops.current().metadataFileLocation()),
core/src/main/java/org/apache/iceberg/ManifestFiles.java: InputFile file = io.newInputFile(manifest.path());
core/src/main/java/org/apache/iceberg/ManifestFiles.java: InputFile file = io.newInputFile(manifest.path());
core/src/main/java/org/apache/iceberg/ManifestsTable.java: ops.io().newInputFile(location != null ? location : ops.current().metadataFileLocation()),
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java: InputFile toCopy = ops.io().newInputFile(manifest.path());
core/src/main/java/org/apache/iceberg/PartitionsTable.java: io().newInputFile(ops.current().metadataFileLocation()),
core/src/main/java/org/apache/iceberg/PartitionsTable.java: io().newInputFile(ops.current().metadataFileLocation()),
core/src/main/java/org/apache/iceberg/RemoveSnapshots.java: return Avro.read(ops.io().newInputFile(snapshot.manifestListLocation()))
core/src/main/java/org/apache/iceberg/SnapshotsTable.java: ops.io().newInputFile(ops.current().metadataFileLocation()),
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: InputFile metadataFile = fileIO.newInputFile(metadataFileLocation);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r852446064
##########
core/src/main/java/org/apache/iceberg/hadoop/CachingHadoopFileIO.java:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hadoop;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Variant of {@link HadoopFileIO} that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingHadoopFileIO extends HadoopFileIO {
Review Comment:
Why is this specific to Hadoop? Why not cache with other IO?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855599076
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -53,6 +56,57 @@ private CatalogProperties() {
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * When using {@link CachingFileIO} as io-impl, define the internal FileIO that is
+ * wrapped by CachingFileIO.
+ * <p>
+ * Default to {@link HadoopFileIO} if not defined.
+ */
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL = "cache.content.wrapped-io-impl";
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT = HadoopFileIO.class.getName();
Review Comment:
`cache-enabled` is already taken by `CachingCatalog`
https://github.com/apache/iceberg/blob/ed54952b51a6cdc7d2350b8223c276421845a8e2/core/src/main/java/org/apache/iceberg/CatalogProperties.java#L39
Is it OK to enable both `CachingCatalog` and `CachingFileIO` by sharing this property key? Or should I make new key like `file-cache-enabled=true`?
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -53,6 +56,57 @@ private CatalogProperties() {
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * When using {@link CachingFileIO} as io-impl, define the internal FileIO that is
+ * wrapped by CachingFileIO.
+ * <p>
+ * Default to {@link HadoopFileIO} if not defined.
+ */
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL = "cache.content.wrapped-io-impl";
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT = HadoopFileIO.class.getName();
+
+ /**
+ * Controls the duration for which entries in the {@link CachingFileIO} are cached.
+ * <p>
+ * Behavior of specific values of cache.expiration-interval-ms:
+ * <ul>
+ * <li> Negative Values - Caching disabled</li>
+ * <li> Zero - Cache entries expires only if it gets evicted due to memory pressure from
+ * {@link #CONTENT_CACHE_MAX_TOTAL_BYTES} setting.
+ * </li>
+ * <li> Positive Values - Cache entries expire if not accessed via the cache after this many milliseconds</li>
+ * </ul>
+ */
+ public static final String CONTENT_CACHE_EXPIRATION_INTERVAL_MS = "cache.content.expiration-interval-ms";
Review Comment:
I see catalog properties already have key `CACHE_ENABLED` and `CACHE_EXPIRATION_INTERVAL_MS` tied to `CachingCatalog` (which has `Cache<TableIdentifier, Table>` inside). Thus, I prefix this with `CONTENT_` to signify that this is different kind of caching. It is caching the file content (bytes) and not file handle or other java objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1107887040
@rizaon, there are a couple of PRs that should help you with this. #4608 adds `IOUtil.readFully` and `IOUtil.readRemaining` that you can use to fill buffers from an input stream. I've also opened #4623 to add the `ByteBufferInputStream` classes I was talking about. Those should allow you to easily split the data across several small allocations and wrap them in an `InputStream` implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862263567
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
Review Comment:
I hate to bring up alternatives so late in the discussion here, but I feel like maybe caching here isn't the right place. I feel like the ManifestReader might be a better place to initiate the wrapping. For example:
ManifestFiles.java:85
```java
InputFile file = CachingFileIO.wrap(io).newInputFile(manifest.path())
```
I feel like this has the potential to cause some interesting side-effects with metadata tables. You could run into caching on worker nodes or expiration of caches due to high churn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] melin commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
melin commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1131536413
CachedFileIO caches the data accessed for the first time. For the mpp database, it affects the corresponding time of the first access. Can snapshot changes be notified to CachedFileIO? Clean cached data, metadata can be cached for a long time.
All metadata is cached in dremio and updated regularly. Snapshot changes, update time is not timely, avoid reading metadata and cause query delay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1113871628
@rizaon and @rdblue I made a comment in one of the replies above but wanted to ask about potentially bad behaviors around reading metadata tables. Since in those cases the data files are really the metadata files, the caching would trigger. I feel like this could contribute to some interesting problems with high churn in those cases both in the driver and potentially the worker side.
It might be better just to wrap on access where we read metadata files in `MetadataFiles.java`. That would also avoid guessing at the pathing for metadata files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1123867490
Hi @danielcweeks @rdblue , can I get a follow up review on this PR please?
I think we have not fully agree on how to decide which file to cache (regex vs specific callsite). I decided to go with regex approach in [dec17e2](https://github.com/apache/iceberg/pull/4518/commits/dec17e2c7d88b972c7d1c817c2cbe66322482146) because there are many callsites in Iceberg core that open metadata files. Please let me know how I can improve this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r958669856
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +328,14 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ InputFile inputFile = io.newInputFile(path, length);
Review Comment:
`inputFile` is not used as cache key, but rather as a fallback within `CachingInputFile` class (field `wrappedInputFile `) if `path` does not exist in `ContentCache` yet.
I understand that `HadoopInputFile.fromLocation()` will create a new `FileSystem` upon initialization. Maybe I can tweak the code to lazily instantiate `wrappedInputFile` by passing `io`, `path`, and `length` instead as constructor parameters of `CachingInputFile`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] steveloughran commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
steveloughran commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r961769554
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return (buf != null) ? buf.length : wrappedInputFile().getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", location);
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", location);
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
+ try {
+ long fileLength = getLength();
+ long totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile().newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 4MB chunk
+ int bytesToRead = (int) Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // Read less than it should be, possibly hitting EOF.
Review Comment:
if less came back than expected, that's possibly a sign of a serious problem (file overwrite?).
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return (buf != null) ? buf.length : wrappedInputFile().getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", location);
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", location);
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
+ try {
+ long fileLength = getLength();
+ long totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile().newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 4MB chunk
+ int bytesToRead = (int) Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // Read less than it should be, possibly hitting EOF.
+ // Set smaller ByteBuffer limit and break out of the loop.
+ buffers.add(ByteBuffer.wrap(buf, 0, bytesRead));
+ break;
+ } else {
+ buffers.add(ByteBuffer.wrap(buf));
+ }
+ }
+
+ stream.close();
Review Comment:
must be in a finally clause or try-with-resources, otherwise s3a will keep the http connection around.
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return (buf != null) ? buf.length : wrappedInputFile().getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", location);
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", location);
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
+ try {
+ long fileLength = getLength();
+ long totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile().newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
Review Comment:
might be good to log at debug this/its duration
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return (buf != null) ? buf.length : wrappedInputFile().getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", location);
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", location);
Review Comment:
could you add e.toString to the error ext, as there's always the risk that something (hive...) will not print the innermost exception after wrapping things a few more times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976134398
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ durationMs);
+ } else if (totalBytes <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ totalBytes);
+ } else if (contentLength <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ contentLength);
+ } else {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ }
+
+ return null;
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("expireAfterAccessMs", expireAfterAccessMs)
+ .add("maxContentLength", maxContentLength)
+ .add("maxTotalBytes", maxTotalBytes)
+ .add("cacheStats", cache.stats())
+ .toString();
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ if (buf != null) {
+ return buf.length;
+ } else if (fallbackInputFile != null) {
+ return fallbackInputFile.getLength();
+ } else {
+ return length;
+ }
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ } catch (IOException e) {
+ throw new RuntimeIOException(
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976141635
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -44,6 +51,28 @@ private ManifestFiles() {}
ManifestFile.PARTITION_SUMMARY_TYPE,
GenericPartitionFieldSummary.class.getName()));
+ private static final Cache<FileIO, ContentCache> CONTENT_CACHES =
+ Caffeine.newBuilder().weakKeys().maximumSize(maxFileIO()).recordStats().build();
Review Comment:
The [documentation](https://www.javadoc.io/doc/com.github.ben-manes.caffeine/caffeine/2.2.0/com/github/benmanes/caffeine/cache/Caffeine.html) says
> Entries with reclaimed keys or values may be removed from the cache on each cache modification, on occasional cache accesses, or on calls to [Cache.cleanUp()](https://www.javadoc.io/static/com.github.ben-manes.caffeine/caffeine/2.2.0/com/github/benmanes/caffeine/cache/Cache.html#cleanUp--);
Added `testWeakFileIOReferenceCleanUp()` to verify that using `weakKeys()` can still evicts cache entries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1256455762
Hey @rizaon I think this is getting real close now (thanks for sticking with us on this). Now that we've gotten this far, I see there's a little more we can do to consolidate the logic in ContentCache. For example, if you pull the property configuration into ContentCache constructor, we can remove a lot from ManifestFiles. For example:
```java
public ContentCache(FileIO fileio) {
this.fileio = fileio;
long expireAfterAccessMs = PropertyUtil.propertyAsLong(
fileio.properties(),
CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS,
CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
long maxTotalBytes = PropertyUtil.propertyAsLong(
fileio.properties(),
CatalogProperties.IO_MANIFEST_CACHE_MAX_TOTAL_BYTES,
CatalogProperties.IO_MANIFEST_CACHE_MAX_TOTAL_BYTES_DEFAULT);
long maxContentLength = PropertyUtil.propertyAsLong(
fileio.properties(),
CatalogProperties.IO_MANIFEST_CACHE_MAX_CONTENT_LENGTH,
CatalogProperties.IO_MANIFEST_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
ValidationException.check(expireAfterAccessMs >= 0, "expireAfterAccessMs is less than 0");
ValidationException.check(maxTotalBytes > 0, "maxTotalBytes is equal or less than 0");
ValidationException.check(maxContentLength > 0, "maxContentLength is equal or less than 0");
this.expireAfterAccessMs = expireAfterAccessMs;
this.maxTotalBytes = maxTotalBytes;
this.maxContentLength = maxContentLength;
```
You can also pull `CONTENT_CACHES`, `contentCache(FileIO io)`, and `dropCache(FileIO fileIO)` into `ContentCache` as well.
I played around with moving more of the logic into `ContentCache` and was able to move everything (minus the `newInputFile`) logic into ContentCache, which I feel is a cleaner separation of concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1145014691
Hi @danielcweeks , given the immutability of snapshot and metadata, does the "potentially bad behaviors around reading metadata tables" still a concern? Immutability should make consistency easy to manage (can just evict cache entry when space is needed without having to worry about invalidating on update, etc).
About "high churn" in terms of high eviction rates, I'd like to understand what scenario can lead to that issue. Within a single table snapshot, what is the largest amount of manifest files and their individual size have been observed before?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r971531204
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
Review Comment:
AFAIK we tend to use `MoreObjects.toStringHelper(..)` for generating `toString` methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r971536224
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
Review Comment:
How long do we keep this reference for the `InputFile`. Does this have any reference for important resources?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r972092207
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
Review Comment:
Fixed in 99057de
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r972093646
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
Review Comment:
Fixed in 99057de
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855666876
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
+ }
+
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ close(); // close and discard any previous FileIO instances
+ this.properties = newProperties;
+ initCache();
+ initWrappedIO();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.CONTENT_CACHE_SHARED,
+ CatalogProperties.CONTENT_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getSharedCache(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ public Configuration conf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.hadoopConf = new SerializableConfiguration(conf)::get;
+ }
+
+ @Override
+ public Configuration getConf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void serializeConfWith(Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
+ this.hadoopConf = confSerializer.apply(hadoopConf.get());
+ }
+
+ private void initWrappedIO() {
+ String wrappedIOImpl = PropertyUtil.propertyAsString(properties,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT);
+ this.wrappedIO = CatalogUtil.loadFileIO(wrappedIOImpl, properties, conf());
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, byte[]> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, byte[]>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, byte[]> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private byte[] getFileContent() {
+ try {
+ int totalByteRead = 0;
+ boolean eos = false;
+ int fileLength = (int) getLength();
+ byte[] buf = new byte[fileLength];
+ SeekableInputStream stream = wrappedInputFile.newStream();
+ while (totalByteRead < fileLength && !eos) {
+ int byteRead = stream.read(buf, totalByteRead, fileLength - totalByteRead);
+ if (byteRead < 0) {
+ // Somehow we hit EOS and ended up short. Truncate buf.
+ buf = Arrays.copyOf(buf, totalByteRead);
+ eos = true;
+ } else {
+ totalByteRead += byteRead;
+ }
+ }
+
+ stream.close();
+ return buf;
+ } catch (IOException ex) {
+ throw new RuntimeIOException(ex);
+ }
+ }
+
+ private SeekableInputStream cachedStream() throws IOException {
+ try {
+ byte[] buf = contentCache.cache().get(location(), k -> getFileContent());
+ LOG.debug("Cache stats: {}", contentCache.cache().stats());
+ return new SeekableCachedInputStream(new SeekableByteArrayInputStream(buf));
Review Comment:
I think this should use a generic in-memory input stream, rather than one that is specific to caching.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855665113
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
Review Comment:
Iceberg avoids using `get` in method names. In most cases it is unnecessary and not adding value. In other cases it can be replaced with a more specific verb. `getOrCreate` seems to work here, if we end up keeping this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1112641357
@rizaon, the other PRs are in so you should be able to take advantage of `IOUtil.readFully` and the `ByteBufferInputStream` implementations. I hope that helps!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861312567
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
Review Comment:
This is now removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862261753
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
+ try {
+ int fileLength = (int) getLength();
Review Comment:
I don't think this should cast to `int`. The length should be tracked as a `long`. I know that we expect `maxContentLength` to be much shorter, but even that is tracked as a `long`. I think this should just use a `long` instead of risking truncation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862261408
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
Review Comment:
The only place this is used catches `IOException` to re-wrap it in `RuntimeIOException`. How about just calling `getLength()` instead of having this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855666207
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
Review Comment:
Nevermind, I see this hasn't been refactored to pass in the inner `FileIO` yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855664074
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -53,6 +56,57 @@ private CatalogProperties() {
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * When using {@link CachingFileIO} as io-impl, define the internal FileIO that is
+ * wrapped by CachingFileIO.
+ * <p>
+ * Default to {@link HadoopFileIO} if not defined.
+ */
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL = "cache.content.wrapped-io-impl";
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT = HadoopFileIO.class.getName();
Review Comment:
I think this would be an `io.cache-enabled` property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r961815570
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -43,6 +47,30 @@ private ManifestFiles() {}
GenericManifestFile.class.getName(),
ManifestFile.PARTITION_SUMMARY_TYPE,
GenericPartitionFieldSummary.class.getName()));
+ private static final int CONTENT_CACHES_EXPIRATION_INTERVAL_SECOND = 60 * 60; // 1 hour
Review Comment:
It seems like if the content cache lives for only 60s and this cache expires after an hour, we're holding on to a lot of FileIO handles for much longer than necessary.
Is there a reason why this need to be so long? Also it doesn't look like we have an upper bound on this cache, which could be problematic due to how much manifest IO activity can happen during any planning phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r973760205
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ durationMs);
+ } else if (totalBytes <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ totalBytes);
+ } else if (contentLength <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ contentLength);
+ } else {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ }
+
+ return null;
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
Review Comment:
should we include weak keys/values here as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks merged pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks merged PR #4518:
URL: https://github.com/apache/iceberg/pull/4518
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1256816188
In the last github check runs, there was a checkstye issue and exception handling when given `FileIO` does not implement `properties()` method.
I have rebased this PR and add [ba7329d](https://github.com/apache/iceberg/pull/4518/commits/ba7329daffe383ab24b1b778079d31e91944dd5d) to fix this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1258319041
@danielcweeks thank you for accepting this PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r978887917
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +339,58 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ boolean enabled = cachingEnabled(io);
+
+ if (enabled) {
+ ContentCache cache = contentCache(io);
+ Preconditions.checkNotNull(cache);
+ LOG.debug("FileIO-level cache stats: {}", CONTENT_CACHES.stats());
+ return cache.tryCache(io, path, length);
+ }
+
+ // caching is not enable for this io or caught RuntimeException.
+ return io.newInputFile(path, length);
+ }
+
+ private static int maxFileIO() {
+ String value = System.getProperty(SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO);
+ if (value != null) {
+ try {
+ return Integer.parseUnsignedInt(value);
+ } catch (NumberFormatException e) {
+ // will return the default
+ }
+ }
+ return SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO_DEFAULT;
+ }
+
+ public static boolean cachingEnabled(FileIO io) {
Review Comment:
Done.
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +339,58 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ boolean enabled = cachingEnabled(io);
+
+ if (enabled) {
+ ContentCache cache = contentCache(io);
+ Preconditions.checkNotNull(cache);
+ LOG.debug("FileIO-level cache stats: {}", CONTENT_CACHES.stats());
+ return cache.tryCache(io, path, length);
+ }
+
+ // caching is not enable for this io or caught RuntimeException.
+ return io.newInputFile(path, length);
+ }
+
+ private static int maxFileIO() {
+ String value = System.getProperty(SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO);
+ if (value != null) {
+ try {
+ return Integer.parseUnsignedInt(value);
+ } catch (NumberFormatException e) {
+ // will return the default
+ }
+ }
+ return SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO_DEFAULT;
+ }
+
+ public static boolean cachingEnabled(FileIO io) {
+ return PropertyUtil.propertyAsBoolean(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED,
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED_DEFAULT);
+ }
+
+ public static long cacheDurationMs(FileIO io) {
Review Comment:
Done.
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +339,58 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ boolean enabled = cachingEnabled(io);
+
+ if (enabled) {
+ ContentCache cache = contentCache(io);
+ Preconditions.checkNotNull(cache);
+ LOG.debug("FileIO-level cache stats: {}", CONTENT_CACHES.stats());
+ return cache.tryCache(io, path, length);
+ }
+
+ // caching is not enable for this io or caught RuntimeException.
+ return io.newInputFile(path, length);
+ }
+
+ private static int maxFileIO() {
+ String value = System.getProperty(SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO);
+ if (value != null) {
+ try {
+ return Integer.parseUnsignedInt(value);
+ } catch (NumberFormatException e) {
+ // will return the default
+ }
+ }
+ return SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO_DEFAULT;
+ }
+
+ public static boolean cachingEnabled(FileIO io) {
+ return PropertyUtil.propertyAsBoolean(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED,
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED_DEFAULT);
+ }
+
+ public static long cacheDurationMs(FileIO io) {
+ return PropertyUtil.propertyAsLong(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ }
+
+ public static long cacheTotalBytes(FileIO io) {
Review Comment:
Done.
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +339,58 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ boolean enabled = cachingEnabled(io);
+
+ if (enabled) {
+ ContentCache cache = contentCache(io);
+ Preconditions.checkNotNull(cache);
+ LOG.debug("FileIO-level cache stats: {}", CONTENT_CACHES.stats());
+ return cache.tryCache(io, path, length);
+ }
+
+ // caching is not enable for this io or caught RuntimeException.
+ return io.newInputFile(path, length);
+ }
+
+ private static int maxFileIO() {
+ String value = System.getProperty(SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO);
+ if (value != null) {
+ try {
+ return Integer.parseUnsignedInt(value);
+ } catch (NumberFormatException e) {
+ // will return the default
+ }
+ }
+ return SystemProperties.IO_MANIFEST_CACHE_MAX_FILEIO_DEFAULT;
+ }
+
+ public static boolean cachingEnabled(FileIO io) {
+ return PropertyUtil.propertyAsBoolean(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED,
+ CatalogProperties.IO_MANIFEST_CACHE_ENABLED_DEFAULT);
+ }
+
+ public static long cacheDurationMs(FileIO io) {
+ return PropertyUtil.propertyAsLong(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ }
+
+ public static long cacheTotalBytes(FileIO io) {
+ return PropertyUtil.propertyAsLong(
+ io.properties(),
+ CatalogProperties.IO_MANIFEST_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_MANIFEST_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ }
+
+ public static long cacheMaxContentLength(FileIO io) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r962267557
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return (buf != null) ? buf.length : wrappedInputFile().getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", location);
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", location);
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
+ try {
+ long fileLength = getLength();
+ long totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile().newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 4MB chunk
+ int bytesToRead = (int) Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // Read less than it should be, possibly hitting EOF.
+ // Set smaller ByteBuffer limit and break out of the loop.
+ buffers.add(ByteBuffer.wrap(buf, 0, bytesRead));
+ break;
+ } else {
+ buffers.add(ByteBuffer.wrap(buf));
+ }
+ }
+
+ stream.close();
Review Comment:
[f76fa5d](https://github.com/apache/iceberg/pull/4518/commits/f76fa5d7f8257c63fd90df32fd841ac989cb26af) change this by using try-with-resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1238332531
[bb737ab](https://github.com/apache/iceberg/pull/4518/commits/bb737ab85f7f1585d0058d09572e9b965da3bb53) is a rebase + squash of all review so far.
[15d67b3](https://github.com/apache/iceberg/pull/4518/commits/15d67b3c41bada725c566ac3fa14aa71127c17a0) is a fix for kyro serialization failure in `testHadoopFileIOKryoSerialization()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r971529650
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
Review Comment:
Would it worth to add a DEBUG log here why the result is `null`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r850878212
##########
core/src/main/java/org/apache/iceberg/hadoop/HadoopInputFile.java:
##########
@@ -241,4 +272,118 @@ public void setNativeCryptoParameters(NativeFileCryptoParameters nativeCryptoPar
public String toString() {
return path.toString();
}
+
+ protected void setContentCache(CachingHadoopFileIO.ContentCache cache) {
+ this.contentCache = cache;
Review Comment:
We can probably move the cache existence check here and additionally set `length` if appropriate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r852446561
##########
core/src/main/java/org/apache/iceberg/hadoop/ConfigProperties.java:
##########
@@ -26,4 +26,8 @@ private ConfigProperties() {
public static final String ENGINE_HIVE_ENABLED = "iceberg.engine.hive.enabled";
public static final String KEEP_HIVE_STATS = "iceberg.hive.keep.stats";
+ public static final String CACHE_MAX_CONTENT_LENGTH = "iceberg.cache.max-content-length";
Review Comment:
This should not be configured through Hadoop. Instead, it should be configured with catalog properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855444592
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -53,6 +56,57 @@ private CatalogProperties() {
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * When using {@link CachingFileIO} as io-impl, define the internal FileIO that is
+ * wrapped by CachingFileIO.
+ * <p>
+ * Default to {@link HadoopFileIO} if not defined.
+ */
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL = "cache.content.wrapped-io-impl";
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT = HadoopFileIO.class.getName();
+
+ /**
+ * Controls the duration for which entries in the {@link CachingFileIO} are cached.
+ * <p>
+ * Behavior of specific values of cache.expiration-interval-ms:
+ * <ul>
+ * <li> Negative Values - Caching disabled</li>
+ * <li> Zero - Cache entries expires only if it gets evicted due to memory pressure from
+ * {@link #CONTENT_CACHE_MAX_TOTAL_BYTES} setting.
+ * </li>
+ * <li> Positive Values - Cache entries expire if not accessed via the cache after this many milliseconds</li>
+ * </ul>
+ */
+ public static final String CONTENT_CACHE_EXPIRATION_INTERVAL_MS = "cache.content.expiration-interval-ms";
Review Comment:
`content` seems unnecessary. What else would be cached?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r852991487
##########
core/src/main/java/org/apache/iceberg/hadoop/CachingHadoopTables.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hadoop;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.io.FileIO;
+
+public class CachingHadoopTables extends HadoopTables {
Review Comment:
This is now dropped. We are now testing directly against `CachingFileIO`.
##########
core/src/main/java/org/apache/iceberg/hadoop/ConfigProperties.java:
##########
@@ -26,4 +26,8 @@ private ConfigProperties() {
public static final String ENGINE_HIVE_ENABLED = "iceberg.engine.hive.enabled";
public static final String KEEP_HIVE_STATS = "iceberg.hive.keep.stats";
+ public static final String CACHE_MAX_CONTENT_LENGTH = "iceberg.cache.max-content-length";
Review Comment:
Done
##########
core/src/main/java/org/apache/iceberg/hadoop/CachingHadoopTables.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.hadoop;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.io.FileIO;
+
+public class CachingHadoopTables extends HadoopTables {
Review Comment:
This is now dropped. We are now testing directly against `CachingFileIO`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855667527
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
+ }
+
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ close(); // close and discard any previous FileIO instances
+ this.properties = newProperties;
+ initCache();
+ initWrappedIO();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.CONTENT_CACHE_SHARED,
+ CatalogProperties.CONTENT_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getSharedCache(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ public Configuration conf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.hadoopConf = new SerializableConfiguration(conf)::get;
+ }
+
+ @Override
+ public Configuration getConf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void serializeConfWith(Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
+ this.hadoopConf = confSerializer.apply(hadoopConf.get());
+ }
+
+ private void initWrappedIO() {
+ String wrappedIOImpl = PropertyUtil.propertyAsString(properties,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT);
+ this.wrappedIO = CatalogUtil.loadFileIO(wrappedIOImpl, properties, conf());
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, byte[]> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, byte[]>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, byte[]> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private byte[] getFileContent() {
+ try {
+ int totalByteRead = 0;
+ boolean eos = false;
+ int fileLength = (int) getLength();
+ byte[] buf = new byte[fileLength];
Review Comment:
I would prefer not to use a single byte[] allocation. If `getLength` is incorrect (which can evidently be the case for some encrypted streams) then we need to be able to handle that case. Plus, this could easily result in humongous allocations in the JVM that are hard to garbage collect. It's better to allocate ByteBuffer instances as needed and use a stream like Parquet's `ByteBufferInputStream` to stitch them together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855668250
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
+ }
+
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ close(); // close and discard any previous FileIO instances
+ this.properties = newProperties;
+ initCache();
+ initWrappedIO();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.CONTENT_CACHE_SHARED,
+ CatalogProperties.CONTENT_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getSharedCache(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ public Configuration conf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.hadoopConf = new SerializableConfiguration(conf)::get;
+ }
+
+ @Override
+ public Configuration getConf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void serializeConfWith(Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
+ this.hadoopConf = confSerializer.apply(hadoopConf.get());
+ }
+
+ private void initWrappedIO() {
+ String wrappedIOImpl = PropertyUtil.propertyAsString(properties,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT);
+ this.wrappedIO = CatalogUtil.loadFileIO(wrappedIOImpl, properties, conf());
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, byte[]> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, byte[]>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, byte[]> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
Review Comment:
Looks like the intent is to implement a read-through cache. Can you clarify that in the Javadoc for this `FileIO`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862260846
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
Review Comment:
If this is going to reach into `fileContentCache` to call methods on `cache` directly, why have `fileContentCache`? Just to check whether it is null? If it is a useful class, I would expect this to call its API instead of modifying its internal state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862261753
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
+ try {
+ int fileLength = (int) getLength();
Review Comment:
I don't think this should cast to `int`. The length should be tracked as a `long`. I know that we expect `maxContentLength` to be much shorter, but even that is tracked as a `long`. I think this should just use a `long` instead of risking corrupting the length.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861311067
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -53,6 +56,57 @@ private CatalogProperties() {
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * When using {@link CachingFileIO} as io-impl, define the internal FileIO that is
+ * wrapped by CachingFileIO.
+ * <p>
+ * Default to {@link HadoopFileIO} if not defined.
+ */
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL = "cache.content.wrapped-io-impl";
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT = HadoopFileIO.class.getName();
+
+ /**
+ * Controls the duration for which entries in the {@link CachingFileIO} are cached.
+ * <p>
+ * Behavior of specific values of cache.expiration-interval-ms:
+ * <ul>
+ * <li> Negative Values - Caching disabled</li>
+ * <li> Zero - Cache entries expires only if it gets evicted due to memory pressure from
+ * {@link #CONTENT_CACHE_MAX_TOTAL_BYTES} setting.
+ * </li>
+ * <li> Positive Values - Cache entries expire if not accessed via the cache after this many milliseconds</li>
+ * </ul>
+ */
+ public static final String CONTENT_CACHE_EXPIRATION_INTERVAL_MS = "cache.content.expiration-interval-ms";
Review Comment:
I renamed these with `IO_CACHE_` and `io.cache.` prefix to match with `io.cache-enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1176853748
Hey @rizaon, some recent changes actually exposed FileIO properties, which I believe would simplify how we can cache manifest files.
Where we [read manifest files](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/ManifestFiles.java#L85), we now have the file legnth and properties from the FileIO. It seems like we could just add a simple cache to the ManifestFiles and allow caching configuration via `FileIO::properties()`.
I feel like that would really simplify this and avoid depending on things like regex+pathing to isolate which files to cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1183565014
@danielcweeks [4d4c4fa](https://github.com/apache/iceberg/commit/4d4c4fac0ab7b5c99aeaa3c6d3fb03d9dffa27c9) implement the manifest file cache in ManifestFiles.java. Since ManifestFiles is stateless and we might deal with multiple FileIO, the cache is implemented as `Cache<FileIO, ContentCache>` (nested cache). It expire a FileIO's ContentCache after 1 hour of inactivity or `dropCache(FileIO)` is called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1258283494
@rizaon Thanks for your contribution here and keeping this updated! I'm excited to see how this works out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976134187
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ durationMs);
+ } else if (totalBytes <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ totalBytes);
+ } else if (contentLength <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ contentLength);
+ } else {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ }
+
+ return null;
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("expireAfterAccessMs", expireAfterAccessMs)
+ .add("maxContentLength", maxContentLength)
+ .add("maxTotalBytes", maxTotalBytes)
+ .add("cacheStats", cache.stats())
+ .toString();
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ if (buf != null) {
+ return buf.length;
+ } else if (fallbackInputFile != null) {
+ return fallbackInputFile.getLength();
+ } else {
+ return length;
+ }
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ } catch (IOException e) {
+ throw new RuntimeIOException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
+ long start = System.currentTimeMillis();
+ try (SeekableInputStream stream = wrappedInputFile().newStream()) {
+ long fileLength = getLength();
+ long totalBytesToRead = fileLength;
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 4MB chunk
+ int bytesToRead = (int) Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // Read less than it should be, possibly hitting EOF. Abandon caching by throwing
+ // IOException and let the caller fallback to non-caching input file.
+ throw new IOException(
+ "Expected to read "
+ + fileLength
+ + " bytes, but only "
+ + (fileLength - totalBytesToRead)
+ + " bytes "
+ + "read.");
+ } else {
+ buffers.add(ByteBuffer.wrap(buf));
+ }
+ }
+
+ CacheEntry newEntry = new CacheEntry(fileLength - totalBytesToRead, buffers);
+ LOG.debug("cacheEntry took {} ms for {}", (System.currentTimeMillis() - start), location);
+ return newEntry;
+ } catch (IOException ex) {
+ throw new RuntimeIOException(ex);
Review Comment:
Done.
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ durationMs);
+ } else if (totalBytes <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ totalBytes);
+ } else if (contentLength <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ contentLength);
+ } else {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ }
+
+ return null;
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return MoreObjects.toStringHelper(this)
+ .add("expireAfterAccessMs", expireAfterAccessMs)
+ .add("maxContentLength", maxContentLength)
+ .add("maxTotalBytes", maxTotalBytes)
+ .add("cacheStats", cache.stats())
+ .toString();
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ if (buf != null) {
+ return buf.length;
+ } else if (fallbackInputFile != null) {
+ return fallbackInputFile.getLength();
+ } else {
+ return length;
+ }
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or
+ * through the inner FileIO.
+ *
+ * <p>If data file is not cached yet, and it can fit in the cache, the file content will be
+ * cached first before returning a {@link ByteBufferInputStream}. Otherwise, return a new
+ * SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache.
+ * Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to
+ // cache.
+ if (getLength() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile().newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ } catch (IOException e) {
+ throw new RuntimeIOException(
+ e, "Failed to open input stream for file %s: %s", location, e.toString());
+ }
+ }
+
+ @Override
+ public String location() {
+ return location;
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.getIfPresent(location);
+ return buf != null || wrappedInputFile().exists();
+ }
+
+ private CacheEntry cacheEntry() {
+ long start = System.currentTimeMillis();
+ try (SeekableInputStream stream = wrappedInputFile().newStream()) {
+ long fileLength = getLength();
+ long totalBytesToRead = fileLength;
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 4MB chunk
+ int bytesToRead = (int) Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // Read less than it should be, possibly hitting EOF. Abandon caching by throwing
+ // IOException and let the caller fallback to non-caching input file.
+ throw new IOException(
+ "Expected to read "
+ + fileLength
+ + " bytes, but only "
+ + (fileLength - totalBytesToRead)
+ + " bytes "
+ + "read.");
+ } else {
+ buffers.add(ByteBuffer.wrap(buf));
+ }
+ }
+
+ CacheEntry newEntry = new CacheEntry(fileLength - totalBytesToRead, buffers);
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976131824
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +329,30 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ try {
+ ContentCache cache = getOrCreateCache(io);
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1256480398
Hi @danielcweeks that looks like a nice cleanup. But I have 2 concerns:
1. By pushing the config verification down to `ContentCache` constructor, that will make `ContentCache` less generic and more specific towards manifest caching use. Is that OK?
2. If we keep `fileio` as a class member of `ContentCache`, that becomes a strong reference to `fileio`. Thus, the `weakKeys()` in `CONTENT_CACHES` is not useful anymore because there is always a strong reference to that `FileIO` key living in `ContentCache` object (the value)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1149113179
One way to avoid churn / pressure in cache is to detect what is the reason of most recent cache eviction. Say, if CachingFileIO tend to evict most entries in short time due to total memory size constraint, CachingFileIO should consider just bypassing cache for some period or until some entries evicted due to expiration rather than size limit.
We can do that with caffeine cache by implementing RemovalListener
https://www.javadoc.io/doc/com.github.ben-manes.caffeine/caffeine/2.3.1/com/github/benmanes/caffeine/cache/RemovalListener.html
Another way is to impose high watermark limit on cache total weight. Thus, bypassing the cache once this high watermark limit exceeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r861315212
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.ByteArrayInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.hadoop.SerializableConfiguration;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.SerializableSupplier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * This FileIO intend to speedup scan planning by caching most of the table metadata contents in memory and reduce
+ * remote reads.
+ */
+public class CachingFileIO implements FileIO, HadoopConfigurable {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.CONTENT_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.CONTENT_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.CONTENT_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getSharedCache(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ private SerializableSupplier<Configuration> hadoopConf;
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * Constructor used for dynamic FileIO loading.
+ * <p>
+ * {@link Configuration Hadoop configuration} must be set through {@link HadoopFileIO#setConf(Configuration)}
+ */
+ public CachingFileIO() {
+ }
+
+ public CachingFileIO(Configuration hadoopConf) {
+ this(new SerializableConfiguration(hadoopConf)::get);
+ }
+
+ public CachingFileIO(SerializableSupplier<Configuration> hadoopConf) {
+ this.hadoopConf = hadoopConf;
+ }
+
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ close(); // close and discard any previous FileIO instances
+ this.properties = newProperties;
+ initCache();
+ initWrappedIO();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.CONTENT_CACHE_SHARED,
+ CatalogProperties.CONTENT_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getSharedCache(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ public Configuration conf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.hadoopConf = new SerializableConfiguration(conf)::get;
+ }
+
+ @Override
+ public Configuration getConf() {
+ return hadoopConf.get();
+ }
+
+ @Override
+ public void serializeConfWith(Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
+ this.hadoopConf = confSerializer.apply(hadoopConf.get());
+ }
+
+ private void initWrappedIO() {
+ String wrappedIOImpl = PropertyUtil.propertyAsString(properties,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL,
+ CatalogProperties.CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT);
+ this.wrappedIO = CatalogUtil.loadFileIO(wrappedIOImpl, properties, conf());
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, byte[]> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, byte[]>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, byte[]> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ byte[] buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private byte[] getFileContent() {
+ try {
+ int totalByteRead = 0;
+ boolean eos = false;
+ int fileLength = (int) getLength();
+ byte[] buf = new byte[fileLength];
+ SeekableInputStream stream = wrappedInputFile.newStream();
+ while (totalByteRead < fileLength && !eos) {
Review Comment:
Refactored to use `IOUtil.readRemaining`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r958600742
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -300,4 +328,14 @@ private static ManifestFile copyManifestInternal(
return writer.toManifestFile();
}
+
+ private static InputFile newInputFile(FileIO io, String path, long length) {
+ InputFile inputFile = io.newInputFile(path, length);
Review Comment:
I have seen cases where calling `newInputFile` was the bottleneck, since the HadoopFileIo created a new `FileSystem` object every time. Why did we decide to key the cache with the `inputFile` instead of the `path`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r971588311
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (enabled && durationMs >= 0 && totalBytes > 0 && contentLength > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
+ .weigher(
+ (Weigher<String, CacheEntry>)
+ (key, value) -> (int) Math.min(value.length, Integer.MAX_VALUE))
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ public CacheEntry get(String key, Function<String, CacheEntry> mappingFunction) {
+ return cache.get(key, mappingFunction);
+ }
+
+ public CacheEntry getIfPresent(String location) {
+ return cache.getIfPresent(location);
+ }
+
+ public InputFile tryCache(FileIO io, String location, long length) {
+ if (length <= maxContentLength) {
+ return new CachingInputFile(this, io, location, length);
+ }
+ return io.newInputFile(location, length);
+ }
+
+ public void invalidate(String key) {
+ cache.invalidate(key);
+ }
+
+ public void invalidateAll() {
+ cache.invalidateAll();
+ }
+
+ public void cleanUp() {
+ cache.cleanUp();
+ }
+
+ public long estimatedCacheSize() {
+ return cache.estimatedSize();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName()
+ + '{'
+ + "expireAfterAccessMs="
+ + expireAfterAccessMs
+ + ", "
+ + "maxContentLength="
+ + maxContentLength
+ + ", "
+ + "maxTotalBytes="
+ + maxTotalBytes
+ + ", "
+ + "cacheStats="
+ + cache.stats()
+ + '}';
+ }
+
+ private static class CacheEntry {
+ private final long length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(long length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final FileIO io;
+ private final String location;
+ private final long length;
+ private InputFile fallbackInputFile = null;
+
+ private CachingInputFile(ContentCache cache, FileIO io, String location, long length) {
+ this.contentCache = cache;
+ this.io = io;
+ this.location = location;
+ this.length = length;
+ }
+
+ private InputFile wrappedInputFile() {
+ if (fallbackInputFile == null) {
+ fallbackInputFile = io.newInputFile(location, length);
+ }
+ return fallbackInputFile;
Review Comment:
This `InputFile` reference is kept as long as the enclosing `CachingInputFile`. The `CachingInputFile` itself is not cached. Only the byte buffers of the input stream are cached on the first call to `CachingInputFile#cachedStream()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976134008
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -44,6 +51,28 @@ private ManifestFiles() {}
ManifestFile.PARTITION_SUMMARY_TYPE,
GenericPartitionFieldSummary.class.getName()));
+ private static final Cache<FileIO, ContentCache> CONTENT_CACHES =
+ Caffeine.newBuilder().weakKeys().maximumSize(maxFileIO()).recordStats().build();
+
+ /**
+ * Get or create a manifest file cache for a given FileIO.
+ *
+ * <p>Returned cache object will be kept in memory until {@link #dropCache(FileIO)} is called.
+ * Return null if the given FileIO properties does not allow caching.
+ *
+ * @param fileIO A FileIO.
+ * @return A manifest file cache or null if the given FileIO properties does not allow caching.
+ */
+ public static ContentCache getOrCreateCache(FileIO fileIO) {
Review Comment:
This is still useful for testing, to gain access to values of `CONTENT_CACHES`. I made this to package private and rename it to `contentCache(FileIO io)`.
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976135354
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
Review Comment:
Added small documentation in `ContentCache`, `ContentCache.tryCache()`, and `CachingInputFile`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976131188
##########
core/src/main/java/org/apache/iceberg/SystemProperties.java:
##########
@@ -33,6 +33,14 @@ private SystemProperties() {}
/** Whether to use the shared worker pool when planning table scans. */
public static final String SCAN_THREAD_POOL_ENABLED = "iceberg.scan.plan-in-worker-pool";
+ /**
+ * Maximum number of distinct {@link org.apache.iceberg.io.FileIO} that is allowed to have
+ * associated {@link org.apache.iceberg.io.ContentCache} in memory at a time.
+ */
+ public static final String IO_CACHE_MAX_FILEIO = "iceberg.io.cache.max-fileio";
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976137247
##########
core/src/main/java/org/apache/iceberg/io/ContentCache.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.MoreObjects;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ContentCache {
+ private static final Logger LOG = LoggerFactory.getLogger(ContentCache.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+
+ public static ContentCache createCache(Map<String, String> properties) {
+ boolean enabled =
+ PropertyUtil.propertyAsBoolean(
+ properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength =
+ PropertyUtil.propertyAsLong(
+ properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (!enabled) {
+ LOG.debug("No ContentCache created. {} is false.", CatalogProperties.IO_CACHE_ENABLED);
+ } else if (durationMs < 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is less than 0.",
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ durationMs);
+ } else if (totalBytes <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ totalBytes);
+ } else if (contentLength <= 0) {
+ LOG.debug(
+ "No ContentCache created. {} ({}) is equal or less than 0.",
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ contentLength);
+ } else {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ }
+
+ return null;
+ }
+
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ public ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache =
+ builder
+ .maximumWeight(maxTotalBytes)
Review Comment:
Added `softValues()` to this builder.
The key still use strong reference to allow key matching by String `equals()` rather than object match (`==`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r976135847
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -44,6 +51,28 @@ private ManifestFiles() {}
ManifestFile.PARTITION_SUMMARY_TYPE,
GenericPartitionFieldSummary.class.getName()));
+ private static final Cache<FileIO, ContentCache> CONTENT_CACHES =
+ Caffeine.newBuilder().weakKeys().maximumSize(maxFileIO()).recordStats().build();
Review Comment:
I think `softValues()` is more appropriate.
Neither `ContentCache` nor `CacheEntry` held for a long time outside its respective Caffeine cache. So using `weakValues()` will always reap them in every gc even when free memory is plenty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] danielcweeks commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
danielcweeks commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1256505449
>
> Hi @danielcweeks that looks like a nice cleanup. But I have 2 concerns:
>
> 1. By pushing the config verification down to `ContentCache` constructor, that will make `ContentCache` less generic and more specific towards manifest caching use. Is that OK?
> 2. If we keep `fileio` as a class member of `ContentCache`, that becomes a strong reference to `fileio`. Thus, the `weakKeys()` in `CONTENT_CACHES` is not useful anymore because there is always a strong reference to that `FileIO` key living in `ContentCache` object (the value)?
Good points. Let me take one more pass (I was just hoping to consolidate more of the logic in ContentCache, but you're probably right that it makes more sense to keep the manifest settings with ManifestFiles).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1095405864
@rizaon, caching in the FileIO layer would be much more general. You could do things like detect that the file size is less than some threshold and cache it in memory, or detect file names under `metadata` and cache those. I highly recommend looking into the FileIO API!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1101486058
Hello @rdblue. This PR is ready for review.
Please let me know if there is any new feedback or request. I'm happy to follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r855444252
##########
core/src/main/java/org/apache/iceberg/CatalogProperties.java:
##########
@@ -53,6 +56,57 @@ private CatalogProperties() {
public static final long CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = TimeUnit.SECONDS.toMillis(30);
public static final long CACHE_EXPIRATION_INTERVAL_MS_OFF = -1;
+ /**
+ * When using {@link CachingFileIO} as io-impl, define the internal FileIO that is
+ * wrapped by CachingFileIO.
+ * <p>
+ * Default to {@link HadoopFileIO} if not defined.
+ */
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL = "cache.content.wrapped-io-impl";
+ public static final String CONTENT_CACHE_WRAPPED_FILE_IO_IMPL_DEFAULT = HadoopFileIO.class.getName();
Review Comment:
I don't think that configuration needs to be this complicated. Instead of replacing the `FileIO` implementation and adding a second dynamic load, I would prefer using `cache-enabled=true` and handling this where we instantiate `FileIO`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862261997
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
+ try {
+ int fileLength = (int) getLength();
+ int totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile.newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
Review Comment:
I think this check is good. You could keep reading buffers until one is under-sized, but then you have to worry about weird cases like when the content fits exactly in some number of buffers. This should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862260597
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
Review Comment:
Style: `get` in a method name should be omitted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862260503
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
Review Comment:
Should this be a config setting since there is a TODO item about it? A regex could be really useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r862262080
##########
core/src/main/java/org/apache/iceberg/io/CachingFileIO.java:
##########
@@ -0,0 +1,355 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.io;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.github.benmanes.caffeine.cache.Weigher;
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.CatalogProperties;
+import org.apache.iceberg.exceptions.NotFoundException;
+import org.apache.iceberg.exceptions.RuntimeIOException;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.util.PropertyUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Implementation of FileIO that adds metadata content caching features.
+ * <p>
+ * CachingFileIO intend to speedup scan planning by caching most of the table metadata files in memory and reduce
+ * remote reads. It is not meant to be dynamically loaded. Instead, set
+ * {@link CatalogProperties#IO_CACHE_ENABLED} to true and use
+ * {@link org.apache.iceberg.CatalogUtil#loadFileIO(String, Map, Object)} to wrap another FileIO with this
+ * CachingFileIO.
+ */
+public class CachingFileIO implements FileIO {
+ private static final Logger LOG = LoggerFactory.getLogger(CachingFileIO.class);
+ private static final int BUFFER_CHUNK_SIZE = 4 * 1024 * 1024; // 4MB
+ private static ContentCache sharedCache;
+
+ private static ContentCache createCache(Map<String, String> properties) {
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+ long contentLength = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH,
+ CatalogProperties.IO_CACHE_MAX_CONTENT_LENGTH_DEFAULT);
+
+ if (durationMs >= 0 && totalBytes > 0) {
+ return new ContentCache(durationMs, totalBytes, contentLength);
+ } else {
+ return null;
+ }
+ }
+
+ public static synchronized ContentCache getOrCreate(Map<String, String> properties) {
+ if (sharedCache == null) {
+ sharedCache = createCache(properties);
+ }
+ return sharedCache;
+ }
+
+ public static boolean ioCacheEnabled(Map<String, String> properties) {
+ boolean enabled = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_ENABLED,
+ CatalogProperties.IO_CACHE_ENABLED_DEFAULT);
+ long durationMs = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS,
+ CatalogProperties.IO_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT);
+ long totalBytes = PropertyUtil.propertyAsLong(properties,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES,
+ CatalogProperties.IO_CACHE_MAX_TOTAL_BYTES_DEFAULT);
+
+ return enabled && durationMs >= 0 && totalBytes > 0;
+ }
+
+ private ContentCache fileContentCache;
+ private Map<String, String> properties;
+ private FileIO wrappedIO;
+
+ /**
+ * CachingFileIO is not meant to be dynamically loaded.
+ */
+ private CachingFileIO() {
+ }
+
+ public CachingFileIO(FileIO io) {
+ assignIO(io);
+ }
+
+ public void assignIO(FileIO io) {
+ this.wrappedIO = io;
+ }
+
+ /**
+ * Initialize File IO from catalog properties.
+ * <p>
+ * It assumes that the inner FileIO is already initialized and therefore does not reinitialize it.
+ * If this is a reinitialization, it is the caller responsibility to call {@link #close()} and
+ * {@link #assignIO(FileIO)} prior to reinitialization to close the previously assigned FileIO and reassign a new one.
+ * Otherwise, previously assigned FileIO will be kept in use.
+ * <p>
+ * Reinitialization will discard previous content cache unless shared cache is being used
+ * ({@link CatalogProperties#IO_CACHE_SHARED} stays true).
+ * @param newProperties catalog properties
+ */
+ @Override
+ public void initialize(Map<String, String> newProperties) {
+ this.properties = newProperties;
+ initCache();
+ }
+
+ private void initCache() {
+ boolean isSharedCache = PropertyUtil.propertyAsBoolean(properties,
+ CatalogProperties.IO_CACHE_SHARED,
+ CatalogProperties.IO_CACHE_SHARED_DEFAULT);
+ if (isSharedCache) {
+ this.fileContentCache = getOrCreate(properties);
+ LOG.info("CachingFileIO created with shared cache. {} ", this.fileContentCache);
+ } else {
+ this.fileContentCache = createCache(properties);
+ LOG.info("CachingFileIO created. {} ", this.fileContentCache);
+ }
+ }
+
+ @Override
+ public void close() {
+ if (wrappedIO != null) {
+ wrappedIO.close();
+ wrappedIO = null;
+ }
+ }
+
+ @Override
+ public InputFile newInputFile(String path) {
+ InputFile inFile = wrappedIO.newInputFile(path);
+ if (fileContentCache != null && path.contains("/metadata/")) {
+ // TODO: Currently we just believe that this is a metadata file if it has "metadata" dir on its path. But metadata
+ // location can be set differently for different tables. A table might have different metadata path by
+ // setting custom "write.metadata.path" TableProperties.
+ // We might need to extend FileIO interface. Add method 'newMetadataInputFile' and let caller use it to
+ // differentiate metadata files vs other files.
+ return new CachingInputFile(fileContentCache, inFile);
+ }
+ return inFile;
+ }
+
+ @Override
+ public OutputFile newOutputFile(String path) {
+ return wrappedIO.newOutputFile(path);
+ }
+
+ @Override
+ public void deleteFile(String path) {
+ wrappedIO.deleteFile(path);
+ if (fileContentCache != null) {
+ fileContentCache.cache.invalidate(path);
+ }
+ }
+
+ public ContentCache getCache() {
+ return fileContentCache;
+ }
+
+ public long cacheEstimatedSize() {
+ return fileContentCache == null ? 0 : fileContentCache.cache.estimatedSize();
+ }
+
+ private static class CacheEntry {
+ private final int length;
+ private final List<ByteBuffer> buffers;
+
+ private CacheEntry(int length, List<ByteBuffer> buffers) {
+ this.length = length;
+ this.buffers = buffers;
+ }
+ }
+
+ public static class ContentCache {
+ private final long expireAfterAccessMs;
+ private final long maxTotalBytes;
+ private final long maxContentLength;
+ private final Cache<String, CacheEntry> cache;
+
+ private ContentCache(long expireAfterAccessMs, long maxTotalBytes, long maxContentLength) {
+ this.expireAfterAccessMs = expireAfterAccessMs;
+ this.maxTotalBytes = maxTotalBytes;
+ this.maxContentLength = maxContentLength;
+
+ Caffeine<Object, Object> builder = Caffeine.newBuilder();
+ if (expireAfterAccessMs > 0) {
+ builder = builder.expireAfterAccess(Duration.ofMillis(expireAfterAccessMs));
+ }
+
+ this.cache = builder.maximumWeight(maxTotalBytes)
+ .weigher((Weigher<String, CacheEntry>) (key, value) -> value.length)
+ .recordStats()
+ .build();
+ }
+
+ public long expireAfterAccess() {
+ return expireAfterAccessMs;
+ }
+
+ public long maxContentLength() {
+ return maxContentLength;
+ }
+
+ public long maxTotalBytes() {
+ return maxTotalBytes;
+ }
+
+ public Cache<String, CacheEntry> cache() {
+ return cache;
+ }
+
+ public CacheStats stats() {
+ return cache.stats();
+ }
+
+ @Override
+ public String toString() {
+ return getClass().getSimpleName() + '{' +
+ "expireAfterAccessMs=" + expireAfterAccessMs + ", " +
+ "maxContentLength=" + maxContentLength + ", " +
+ "maxTotalBytes=" + maxTotalBytes + ", " +
+ "cacheStats=" + cache.stats() + '}';
+ }
+ }
+
+ private static class CachingInputFile implements InputFile {
+ private final ContentCache contentCache;
+ private final InputFile wrappedInputFile;
+
+ private CachingInputFile(ContentCache cache, InputFile inFile) {
+ this.contentCache = cache;
+ this.wrappedInputFile = inFile;
+ }
+
+ @Override
+ public long getLength() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return (buf != null) ? buf.length : wrappedInputFile.getLength();
+ }
+
+ /**
+ * Opens a new {@link SeekableInputStream} for the underlying data file, either through cache or through the
+ * inner FileIO.
+ * <p>
+ * If data file is not cached yet, and it can fit in the cache, the file content will be cached first before
+ * returning a {@link ByteBufferInputStream}. Otherwise, return a new SeekableInputStream from the inner FIleIO.
+ *
+ * @return a {@link ByteBufferInputStream} if file exist in the cache or can fit in the cache. Otherwise, return a
+ * new SeekableInputStream from the inner FIleIO.
+ */
+ @Override
+ public SeekableInputStream newStream() {
+ try {
+ // read-through cache if file length is less than or equal to maximum length allowed to cache.
+ if (getLengthChecked() <= contentCache.maxContentLength()) {
+ return cachedStream();
+ }
+
+ // fallback to non-caching input stream.
+ return wrappedInputFile.newStream();
+ } catch (FileNotFoundException e) {
+ throw new NotFoundException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ } catch (IOException e) {
+ throw new RuntimeIOException(e, "Failed to open input stream for file: %s", wrappedInputFile.location());
+ }
+ }
+
+ @Override
+ public String location() {
+ return wrappedInputFile.location();
+ }
+
+ @Override
+ public boolean exists() {
+ CacheEntry buf = contentCache.cache.getIfPresent(location());
+ return buf != null || wrappedInputFile.exists();
+ }
+
+ public long getLengthChecked() throws IOException {
+ try {
+ return getLength();
+ } catch (RuntimeIOException ex) {
+ throw ex.getCause();
+ }
+ }
+
+ private CacheEntry getFileContent() {
+ try {
+ int fileLength = (int) getLength();
+ int totalBytesToRead = fileLength;
+ SeekableInputStream stream = wrappedInputFile.newStream();
+ List<ByteBuffer> buffers = Lists.newArrayList();
+
+ while (totalBytesToRead > 0) {
+ // read the stream in 4MB chunk
+ int bytesToRead = Math.min(BUFFER_CHUNK_SIZE, totalBytesToRead);
+ byte[] buf = new byte[bytesToRead];
+ int bytesRead = IOUtil.readRemaining(stream, buf, 0, bytesToRead);
+ totalBytesToRead -= bytesRead;
+
+ if (bytesRead < bytesToRead) {
+ // We are reading less than we should, possibly hitting EOF.
+ // Truncate buffer and break from the loop. In worst case, this will do byte copying of slightly under 4MB.
+ byte[] smallerBuf = new byte[bytesRead];
+ System.arraycopy(buf, 0, smallerBuf, 0, bytesRead);
Review Comment:
Looks like this wasn't updated yet -- I'll come back this weekend after you've had a chance to update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on a diff in pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r865446838
##########
core/src/main/java/org/apache/iceberg/CatalogUtil.java:
##########
@@ -272,6 +273,13 @@ public static FileIO loadFileIO(
configureHadoopConf(fileIO, hadoopConf);
fileIO.initialize(properties);
+
+ // Check if we should wrap this with CachingFileIO
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rizaon commented on pull request #4518: core: Provide mechanism to cache manifest file content
Posted by GitBox <gi...@apache.org>.
rizaon commented on PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#issuecomment-1216706722
Hello @danielcweeks, can I get a new feedback from my latest rebase please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org