You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/08/31 16:57:38 UTC
[GitHub] [pinot] klsince opened a new pull request, #9306: MultiDirSegmentDirectoryLoader to keep segments in multi-datadir
klsince opened a new pull request, #9306:
URL: https://github.com/apache/pinot/pull/9306
This PR is part of the work to support multi-datadir for Pinot server as tracked by [issue](https://github.com/apache/pinot/issues/8843).
MultiDirSegmentDirectoryLoader is added in this PR. It implements SegmentDirectoryLoader interface. It checks the target storage tier for the segment and move it from current tier to the target tier. The tier config is extended to include dataDir, so that one can define multi-tier for a table to use multiple datadirs to store segments.
Following PRs will add Controller periodic task to compare segment current/target tiers and trigger segment reloading to move them to target tiers (datadirs) on Pinot servers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on PR #9306:
URL: https://github.com/apache/pinot/pull/9306#issuecomment-1254351740
Using a tier track file kept in the default segment dir to help find local segment and reuse them as much as possible. pls help take a look @Jackie-Jiang . I'm thinking to put the track file into `tableDataDir/tmp/` directory to be cleaner, lemme know your thought.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9306: MultiDirSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r962069418
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/config/TierConfigUtils.java:
##########
@@ -47,6 +51,28 @@ public static boolean shouldRelocateToTiers(TableConfig tableConfig) {
return CollectionUtils.isNotEmpty(tableConfig.getTierConfigsList());
}
+ public static String getDataDirFromTierConfig(String tableNameWithType, String tierName, TableConfig tableConfig) {
+ List<TierConfig> tierCfgs = tableConfig.getTierConfigsList();
+ Preconditions
+ .checkState(tierCfgs != null && !tierCfgs.isEmpty(), "No tierConfigs for table: %s", tableNameWithType);
Review Comment:
(minor) `CollectionUtils.isNotEmpty(tierCfgs)`
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/loader/MultiDirSegmentDirectoryLoader.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.loader;
+
+import java.io.File;
+import java.net.URI;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.utils.config.TierConfigUtils;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.store.SegmentLocalFSDirectory;
+import org.apache.pinot.segment.spi.loader.SegmentDirectoryLoader;
+import org.apache.pinot.segment.spi.loader.SegmentDirectoryLoaderContext;
+import org.apache.pinot.segment.spi.loader.SegmentLoader;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Implementation of {@link SegmentDirectoryLoader} that can move segments across data dirs configured as storage tiers.
+ */
+@SegmentLoader(name = "multidir")
+public class MultiDirSegmentDirectoryLoader implements SegmentDirectoryLoader {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MultiDirSegmentDirectoryLoader.class);
+
+ /**
+ * Creates and loads the {@link SegmentLocalFSDirectory} which is the default implementation of
+ * {@link SegmentDirectory}
+ * @param indexDir the current segment index directory
+ * @param segmentLoaderContext context for instantiation of the SegmentDirectory. The target tier set in context is
+ * used to decide which data directory to keep the segment data.
+ * @return instance of {@link SegmentLocalFSDirectory}
+ */
+ @Override
+ public SegmentDirectory load(URI indexDir, SegmentDirectoryLoaderContext segmentLoaderContext)
+ throws Exception {
+ File srcDir = new File(indexDir);
+ String segmentName = segmentLoaderContext.getSegmentName();
+ String segmentTier = segmentLoaderContext.getSegmentTier();
+ File destDir = getTargetDataDir(segmentTier, segmentLoaderContext);
+ if (destDir == null) {
+ destDir = getDefaultDataDir(segmentLoaderContext);
+ LOGGER.info("Use destDir: {} on default tier for segment: {} as no dataDir for target tier: {}", destDir,
+ segmentName, segmentTier);
+ segmentTier = null;
+ }
+ if (srcDir.equals(destDir)) {
+ LOGGER.info("Keep segment: {} in current dataDir: {} on current tier: {}", segmentName, destDir, segmentTier);
+ } else {
+ LOGGER.info("Move segment: {} from srcDir: {} to destDir: {} on tier: {}", segmentName, srcDir, destDir,
+ segmentTier);
+ if (destDir.exists()) {
+ LOGGER.warn("The destDir: {} exists on tier: {} and cleans it firstly", destDir, segmentTier);
+ FileUtils.deleteQuietly(destDir);
+ }
+ FileUtils.moveDirectory(srcDir, destDir);
Review Comment:
Will this cause exception when `srcDir` does not exist?
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -546,6 +553,31 @@ File getSegmentDataDir(String segmentName) {
return new File(_indexDir, segmentName);
}
+ @VisibleForTesting
+ File getSegmentDataDir(String segmentName, String segmentTier, IndexLoadingConfig indexLoadingConfig) {
Review Comment:
(minor)
```suggestion
File getSegmentDataDir(String segmentName, @Nullable String segmentTier, IndexLoadingConfig indexLoadingConfig) {
```
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -642,44 +675,44 @@ private boolean tryLoadExistingSegment(String segmentName, IndexLoadingConfig in
// Close the stale SegmentDirectory object and recreate it with reprocessed segment.
closeSegmentDirectoryQuietly(segmentDirectory);
ImmutableSegmentLoader.preprocess(indexDir, indexLoadingConfig, schema);
- segmentDirectory =
- initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()), indexLoadingConfig, schema);
+ segmentDirectory = initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()), zkMetadata.getTier(),
+ indexLoadingConfig, schema);
}
ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDirectory, indexLoadingConfig, schema);
addSegment(segment);
- LOGGER.info("Loaded existing segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc());
+ LOGGER.info("Loaded existing segment: {} of table: {} with crc: {} on tier: {}", segmentName, _tableNameWithType,
+ zkMetadata.getCrc(), zkMetadata.getTier());
return true;
} catch (Exception e) {
- LOGGER.error("Failed to load existing segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc(), e);
+ LOGGER.error("Failed to load existing segment: {} of table: {} with crc: {} on tier: {}", segmentName,
+ _tableNameWithType, zkMetadata.getCrc(), zkMetadata.getTier(), e);
closeSegmentDirectoryQuietly(segmentDirectory);
return false;
}
}
- private SegmentDirectory tryInitSegmentDirectory(String segmentName, String segmentCrc,
+ private SegmentDirectory tryInitSegmentDirectory(String segmentName, String segmentCrc, String segmentTier,
IndexLoadingConfig indexLoadingConfig, Schema schema) {
try {
- return initSegmentDirectory(segmentName, segmentCrc, indexLoadingConfig, schema);
+ return initSegmentDirectory(segmentName, segmentCrc, segmentTier, indexLoadingConfig, schema);
} catch (Exception e) {
- LOGGER.warn("Failed to initialize SegmentDirectory for segment: {} of table: {} with error: {}", segmentName,
- _tableNameWithType, e.getMessage());
+ LOGGER.warn("Failed to initialize SegmentDirectory for segment: {} of table: {} on tier: {} with error: {}",
+ segmentName, _tableNameWithType, segmentTier, e.getMessage());
return null;
}
}
- private SegmentDirectory initSegmentDirectory(String segmentName, String segmentCrc,
+ private SegmentDirectory initSegmentDirectory(String segmentName, String segmentCrc, String segmentTier,
IndexLoadingConfig indexLoadingConfig, Schema schema)
throws Exception {
SegmentDirectoryLoaderContext loaderContext =
new SegmentDirectoryLoaderContext.Builder().setTableConfig(indexLoadingConfig.getTableConfig())
- .setSchema(schema).setInstanceId(indexLoadingConfig.getInstanceId()).setSegmentName(segmentName)
- .setSegmentCrc(segmentCrc).setSegmentDirectoryConfigs(indexLoadingConfig.getSegmentDirectoryConfigs())
- .build();
+ .setSchema(schema).setInstanceId(indexLoadingConfig.getInstanceId()).setTableDataDir(_tableDataDir)
+ .setSegmentName(segmentName).setSegmentCrc(segmentCrc).setSegmentTier(segmentTier)
+ .setSegmentDirectoryConfigs(indexLoadingConfig.getSegmentDirectoryConfigs()).build();
SegmentDirectoryLoader segmentDirectoryLoader =
SegmentDirectoryLoaderRegistry.getSegmentDirectoryLoader(indexLoadingConfig.getSegmentDirectoryLoader());
- File indexDir = getSegmentDataDir(segmentName);
+ File indexDir = getSegmentDataDir(segmentName, segmentTier, indexLoadingConfig);
return segmentDirectoryLoader.load(indexDir.toURI(), loaderContext);
Review Comment:
I'm I missing something? The source dir here will be the final dir after applying the tier, but the file might not be moved yet (the move logic is within the load). Won't this cause chicken and egg problem?
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/loader/SegmentDirectoryLoaderContext.java:
##########
@@ -75,6 +87,8 @@ public static class Builder {
private String _instanceId;
private String _segmentName;
private String _segmentCrc;
+ private String _segmentTier;
+ private String _tableDataDir;
Review Comment:
(minor) Keep the same sequence as the constructor
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/loader/MultiDirSegmentDirectoryLoader.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.loader;
+
+import java.io.File;
+import java.net.URI;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.utils.config.TierConfigUtils;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.store.SegmentLocalFSDirectory;
+import org.apache.pinot.segment.spi.loader.SegmentDirectoryLoader;
+import org.apache.pinot.segment.spi.loader.SegmentDirectoryLoaderContext;
+import org.apache.pinot.segment.spi.loader.SegmentLoader;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Implementation of {@link SegmentDirectoryLoader} that can move segments across data dirs configured as storage tiers.
+ */
+@SegmentLoader(name = "multidir")
+public class MultiDirSegmentDirectoryLoader implements SegmentDirectoryLoader {
Review Comment:
Should we consider calling it `TierBasedSegmentDirectoryLoader`
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -406,9 +410,12 @@ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoad
localMetadata.getCrc(), zkMetadata.getCrc());
}
File indexDir = downloadSegment(segmentName, zkMetadata);
- addSegment(indexDir, indexLoadingConfig);
- LOGGER.info("Downloaded and loaded segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc());
+ Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore, _tableNameWithType);
Review Comment:
We might want to remove the `addSegment(indexDir, indexLoadingConfig)` since it doesn't have all the required arguments, and can cause unexpected behavior when people use it
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -546,6 +553,31 @@ File getSegmentDataDir(String segmentName) {
return new File(_indexDir, segmentName);
}
+ @VisibleForTesting
+ File getSegmentDataDir(String segmentName, String segmentTier, IndexLoadingConfig indexLoadingConfig) {
+ if (segmentTier == null) {
+ return getSegmentDataDir(segmentName);
+ }
+ try {
+ String tierDataDir = TierConfigUtils
+ .getDataDirFromTierConfig(_tableNameWithType, segmentTier, indexLoadingConfig.getTableConfig());
+ File tierTableDataDir = new File(tierDataDir, _tableNameWithType);
+ return new File(tierTableDataDir, segmentName);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to get dataDir for segment: {} of table: {} on tier: {} due to error: {}", segmentName,
+ _tableNameWithType, segmentTier, e.getMessage());
+ return getSegmentDataDir(segmentName);
+ }
+ }
+
+ private String getSegmentCurrentTier(String segmentName) {
Review Comment:
(minor) Annotate the return as `Nullable`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r995111114
##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentOnlineOfflineStateModelFactory.java:
##########
@@ -200,22 +196,11 @@ public void onBecomeDroppedFromOffline(Message message, NotificationContext cont
_logger.info("SegmentOnlineOfflineStateModel.onBecomeDroppedFromOffline() : " + message);
String tableNameWithType = message.getResourceName();
String segmentName = message.getPartitionName();
-
- // This method might modify the file on disk. Use segment lock to prevent race condition
- Lock segmentLock = SegmentLocks.getSegmentLock(tableNameWithType, segmentName);
Review Comment:
We still need to lock the segment because it will modify the file
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/InstanceDataManager.java:
##########
@@ -86,11 +86,17 @@ void addRealtimeSegment(String realtimeTableName, String segmentName)
throws Exception;
/**
- * Removes a segment from a table.
+ * Removes a segment from a table but not dropping its data from server.
*/
void removeSegment(String tableNameWithType, String segmentName)
throws Exception;
+ /**
+ * Delete segment data from the server.
+ */
+ void dropSegment(String tableNameWithType, String segmentName)
Review Comment:
Consider renaming to `deleteSegmentData` to be more specific
##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java:
##########
@@ -226,6 +230,45 @@ public void removeSegment(String tableNameWithType, String segmentName) {
});
}
+ @Override
+ public void dropSegment(String tableNameWithType, String segmentName)
+ throws Exception {
+ // This method might modify the file on disk. Use segment lock to prevent race condition
+ Lock segmentLock = SegmentLocks.getSegmentLock(tableNameWithType, segmentName);
+ try {
+ segmentLock.lock();
+
+ // Clean up the segment data on default tier unconditionally.
+ File segmentDir = getSegmentDataDirectory(tableNameWithType, segmentName);
+ if (segmentDir.exists()) {
+ FileUtils.deleteQuietly(segmentDir);
+ LOGGER.info("Deleted segment directory {} on default tier", segmentDir);
+ }
+ // Note that this method usually happens after removeSegment, which removes the segment object from the
+ // tableDataManager object already, so we can't check segment info from there. In addition, tableDataManager
+ // object itself might not be present for the given table on the server either. So the locations of segment data
+ // are derived from configs and cached info at best effort.
+ TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
Review Comment:
This won't work when the table is deleted. We should be able to get the tier from the local tier file, and use it to find the data directory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: MultiDirSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r964223310
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -406,9 +410,12 @@ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoad
localMetadata.getCrc(), zkMetadata.getCrc());
}
File indexDir = downloadSegment(segmentName, zkMetadata);
- addSegment(indexDir, indexLoadingConfig);
- LOGGER.info("Downloaded and loaded segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc());
+ Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore, _tableNameWithType);
Review Comment:
actually, we can change it as below so it loads segment to local tier by default, although this method is not used any more.
```
addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema, true, _tableDataDir, null));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r979092597
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -329,7 +330,8 @@ public Map<String, SegmentErrorInfo> getSegmentErrors() {
public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig, SegmentZKMetadata zkMetadata,
SegmentMetadata localMetadata, @Nullable Schema schema, boolean forceDownload)
throws Exception {
- File indexDir = getSegmentDataDir(segmentName);
+ String segmentTier = getSegmentCurrentTier(segmentName);
Review Comment:
I thought about using SegmentMetadataImpl at beginning, but I see its content is all from the metadata.properties file, so I assume it's used as an (effective) immutable object. And MutableSegment has SegmentMetadataImpl object too, but it doesn't have tier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: MultiDirSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r963977208
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/loader/MultiDirSegmentDirectoryLoader.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.loader;
+
+import java.io.File;
+import java.net.URI;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.utils.config.TierConfigUtils;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.store.SegmentLocalFSDirectory;
+import org.apache.pinot.segment.spi.loader.SegmentDirectoryLoader;
+import org.apache.pinot.segment.spi.loader.SegmentDirectoryLoaderContext;
+import org.apache.pinot.segment.spi.loader.SegmentLoader;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Implementation of {@link SegmentDirectoryLoader} that can move segments across data dirs configured as storage tiers.
+ */
+@SegmentLoader(name = "multidir")
+public class MultiDirSegmentDirectoryLoader implements SegmentDirectoryLoader {
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: MultiDirSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r963976978
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/loader/MultiDirSegmentDirectoryLoader.java:
##########
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.loader;
+
+import java.io.File;
+import java.net.URI;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.utils.config.TierConfigUtils;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.store.SegmentLocalFSDirectory;
+import org.apache.pinot.segment.spi.loader.SegmentDirectoryLoader;
+import org.apache.pinot.segment.spi.loader.SegmentDirectoryLoaderContext;
+import org.apache.pinot.segment.spi.loader.SegmentLoader;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Implementation of {@link SegmentDirectoryLoader} that can move segments across data dirs configured as storage tiers.
+ */
+@SegmentLoader(name = "multidir")
+public class MultiDirSegmentDirectoryLoader implements SegmentDirectoryLoader {
+ private static final Logger LOGGER = LoggerFactory.getLogger(MultiDirSegmentDirectoryLoader.class);
+
+ /**
+ * Creates and loads the {@link SegmentLocalFSDirectory} which is the default implementation of
+ * {@link SegmentDirectory}
+ * @param indexDir the current segment index directory
+ * @param segmentLoaderContext context for instantiation of the SegmentDirectory. The target tier set in context is
+ * used to decide which data directory to keep the segment data.
+ * @return instance of {@link SegmentLocalFSDirectory}
+ */
+ @Override
+ public SegmentDirectory load(URI indexDir, SegmentDirectoryLoaderContext segmentLoaderContext)
+ throws Exception {
+ File srcDir = new File(indexDir);
+ String segmentName = segmentLoaderContext.getSegmentName();
+ String segmentTier = segmentLoaderContext.getSegmentTier();
+ File destDir = getTargetDataDir(segmentTier, segmentLoaderContext);
+ if (destDir == null) {
+ destDir = getDefaultDataDir(segmentLoaderContext);
+ LOGGER.info("Use destDir: {} on default tier for segment: {} as no dataDir for target tier: {}", destDir,
+ segmentName, segmentTier);
+ segmentTier = null;
+ }
+ if (srcDir.equals(destDir)) {
+ LOGGER.info("Keep segment: {} in current dataDir: {} on current tier: {}", segmentName, destDir, segmentTier);
+ } else {
+ LOGGER.info("Move segment: {} from srcDir: {} to destDir: {} on tier: {}", segmentName, srcDir, destDir,
+ segmentTier);
+ if (destDir.exists()) {
+ LOGGER.warn("The destDir: {} exists on tier: {} and cleans it firstly", destDir, segmentTier);
+ FileUtils.deleteQuietly(destDir);
+ }
+ FileUtils.moveDirectory(srcDir, destDir);
Review Comment:
yes, it will and it should throw exception as srcDir is assumed to exist. The caller of loader.load() makes sure the provided `indexDir` exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on PR #9306:
URL: https://github.com/apache/pinot/pull/9306#issuecomment-1256723404
> (MAJOR) the segment delete logic won't work with tier data dir....
good point, will check it out.
> As for the tracking file management, I'm thinking to keep a file of name `<segment>.tier` under the table data dir.
I like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r995117637
##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentOnlineOfflineStateModelFactory.java:
##########
@@ -200,22 +196,11 @@ public void onBecomeDroppedFromOffline(Message message, NotificationContext cont
_logger.info("SegmentOnlineOfflineStateModel.onBecomeDroppedFromOffline() : " + message);
String tableNameWithType = message.getResourceName();
String segmentName = message.getPartitionName();
-
- // This method might modify the file on disk. Use segment lock to prevent race condition
- Lock segmentLock = SegmentLocks.getSegmentLock(tableNameWithType, segmentName);
Review Comment:
locked inside dropSegment() method, like the other methods reloadSegment or so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r994953011
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -329,7 +330,8 @@ public Map<String, SegmentErrorInfo> getSegmentErrors() {
public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig, SegmentZKMetadata zkMetadata,
SegmentMetadata localMetadata, @Nullable Schema schema, boolean forceDownload)
throws Exception {
- File indexDir = getSegmentDataDir(segmentName);
+ String segmentTier = getSegmentCurrentTier(segmentName);
Review Comment:
I'll refactor this in a separate PR, as it's orthogonal to the feature of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] npawar commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
npawar commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r967914605
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -642,44 +675,44 @@ private boolean tryLoadExistingSegment(String segmentName, IndexLoadingConfig in
// Close the stale SegmentDirectory object and recreate it with reprocessed segment.
closeSegmentDirectoryQuietly(segmentDirectory);
ImmutableSegmentLoader.preprocess(indexDir, indexLoadingConfig, schema);
- segmentDirectory =
- initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()), indexLoadingConfig, schema);
+ segmentDirectory = initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()), zkMetadata.getTier(),
+ indexLoadingConfig, schema);
}
ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDirectory, indexLoadingConfig, schema);
addSegment(segment);
- LOGGER.info("Loaded existing segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc());
+ LOGGER.info("Loaded existing segment: {} of table: {} with crc: {} on tier: {}", segmentName, _tableNameWithType,
+ zkMetadata.getCrc(), zkMetadata.getTier());
return true;
} catch (Exception e) {
- LOGGER.error("Failed to load existing segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc(), e);
+ LOGGER.error("Failed to load existing segment: {} of table: {} with crc: {} on tier: {}", segmentName,
+ _tableNameWithType, zkMetadata.getCrc(), zkMetadata.getTier(), e);
closeSegmentDirectoryQuietly(segmentDirectory);
return false;
}
}
- private SegmentDirectory tryInitSegmentDirectory(String segmentName, String segmentCrc,
+ private SegmentDirectory tryInitSegmentDirectory(String segmentName, String segmentCrc, String segmentTier,
IndexLoadingConfig indexLoadingConfig, Schema schema) {
try {
- return initSegmentDirectory(segmentName, segmentCrc, indexLoadingConfig, schema);
+ return initSegmentDirectory(segmentName, segmentCrc, segmentTier, indexLoadingConfig, schema);
} catch (Exception e) {
- LOGGER.warn("Failed to initialize SegmentDirectory for segment: {} of table: {} with error: {}", segmentName,
- _tableNameWithType, e.getMessage());
+ LOGGER.warn("Failed to initialize SegmentDirectory for segment: {} of table: {} on tier: {} with error: {}",
+ segmentName, _tableNameWithType, segmentTier, e.getMessage());
return null;
}
}
- private SegmentDirectory initSegmentDirectory(String segmentName, String segmentCrc,
+ private SegmentDirectory initSegmentDirectory(String segmentName, String segmentCrc, String segmentTier,
IndexLoadingConfig indexLoadingConfig, Schema schema)
throws Exception {
SegmentDirectoryLoaderContext loaderContext =
new SegmentDirectoryLoaderContext.Builder().setTableConfig(indexLoadingConfig.getTableConfig())
- .setSchema(schema).setInstanceId(indexLoadingConfig.getInstanceId()).setSegmentName(segmentName)
- .setSegmentCrc(segmentCrc).setSegmentDirectoryConfigs(indexLoadingConfig.getSegmentDirectoryConfigs())
- .build();
+ .setSchema(schema).setInstanceId(indexLoadingConfig.getInstanceId()).setTableDataDir(_tableDataDir)
+ .setSegmentName(segmentName).setSegmentCrc(segmentCrc).setSegmentTier(segmentTier)
+ .setSegmentDirectoryConfigs(indexLoadingConfig.getSegmentDirectoryConfigs()).build();
SegmentDirectoryLoader segmentDirectoryLoader =
SegmentDirectoryLoaderRegistry.getSegmentDirectoryLoader(indexLoadingConfig.getSegmentDirectoryLoader());
- File indexDir = getSegmentDataDir(segmentName);
+ File indexDir = getSegmentDataDir(segmentName, segmentTier, indexLoadingConfig);
return segmentDirectoryLoader.load(indexDir.toURI(), loaderContext);
Review Comment:
minor suggestion: in that case, please add current/target or src/dest prefix to every segmentTier and indexDir variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: MultiDirSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r963980370
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -642,44 +675,44 @@ private boolean tryLoadExistingSegment(String segmentName, IndexLoadingConfig in
// Close the stale SegmentDirectory object and recreate it with reprocessed segment.
closeSegmentDirectoryQuietly(segmentDirectory);
ImmutableSegmentLoader.preprocess(indexDir, indexLoadingConfig, schema);
- segmentDirectory =
- initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()), indexLoadingConfig, schema);
+ segmentDirectory = initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()), zkMetadata.getTier(),
+ indexLoadingConfig, schema);
}
ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDirectory, indexLoadingConfig, schema);
addSegment(segment);
- LOGGER.info("Loaded existing segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc());
+ LOGGER.info("Loaded existing segment: {} of table: {} with crc: {} on tier: {}", segmentName, _tableNameWithType,
+ zkMetadata.getCrc(), zkMetadata.getTier());
return true;
} catch (Exception e) {
- LOGGER.error("Failed to load existing segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc(), e);
+ LOGGER.error("Failed to load existing segment: {} of table: {} with crc: {} on tier: {}", segmentName,
+ _tableNameWithType, zkMetadata.getCrc(), zkMetadata.getTier(), e);
closeSegmentDirectoryQuietly(segmentDirectory);
return false;
}
}
- private SegmentDirectory tryInitSegmentDirectory(String segmentName, String segmentCrc,
+ private SegmentDirectory tryInitSegmentDirectory(String segmentName, String segmentCrc, String segmentTier,
IndexLoadingConfig indexLoadingConfig, Schema schema) {
try {
- return initSegmentDirectory(segmentName, segmentCrc, indexLoadingConfig, schema);
+ return initSegmentDirectory(segmentName, segmentCrc, segmentTier, indexLoadingConfig, schema);
} catch (Exception e) {
- LOGGER.warn("Failed to initialize SegmentDirectory for segment: {} of table: {} with error: {}", segmentName,
- _tableNameWithType, e.getMessage());
+ LOGGER.warn("Failed to initialize SegmentDirectory for segment: {} of table: {} on tier: {} with error: {}",
+ segmentName, _tableNameWithType, segmentTier, e.getMessage());
return null;
}
}
- private SegmentDirectory initSegmentDirectory(String segmentName, String segmentCrc,
+ private SegmentDirectory initSegmentDirectory(String segmentName, String segmentCrc, String segmentTier,
IndexLoadingConfig indexLoadingConfig, Schema schema)
throws Exception {
SegmentDirectoryLoaderContext loaderContext =
new SegmentDirectoryLoaderContext.Builder().setTableConfig(indexLoadingConfig.getTableConfig())
- .setSchema(schema).setInstanceId(indexLoadingConfig.getInstanceId()).setSegmentName(segmentName)
- .setSegmentCrc(segmentCrc).setSegmentDirectoryConfigs(indexLoadingConfig.getSegmentDirectoryConfigs())
- .build();
+ .setSchema(schema).setInstanceId(indexLoadingConfig.getInstanceId()).setTableDataDir(_tableDataDir)
+ .setSegmentName(segmentName).setSegmentCrc(segmentCrc).setSegmentTier(segmentTier)
+ .setSegmentDirectoryConfigs(indexLoadingConfig.getSegmentDirectoryConfigs()).build();
SegmentDirectoryLoader segmentDirectoryLoader =
SegmentDirectoryLoaderRegistry.getSegmentDirectoryLoader(indexLoadingConfig.getSegmentDirectoryLoader());
- File indexDir = getSegmentDataDir(segmentName);
+ File indexDir = getSegmentDataDir(segmentName, segmentTier, indexLoadingConfig);
return segmentDirectoryLoader.load(indexDir.toURI(), loaderContext);
Review Comment:
the provided segmentTier is the `current tier` of the segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r994986826
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -642,44 +675,44 @@ private boolean tryLoadExistingSegment(String segmentName, IndexLoadingConfig in
// Close the stale SegmentDirectory object and recreate it with reprocessed segment.
closeSegmentDirectoryQuietly(segmentDirectory);
ImmutableSegmentLoader.preprocess(indexDir, indexLoadingConfig, schema);
- segmentDirectory =
- initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()), indexLoadingConfig, schema);
+ segmentDirectory = initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()), zkMetadata.getTier(),
+ indexLoadingConfig, schema);
}
ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDirectory, indexLoadingConfig, schema);
addSegment(segment);
- LOGGER.info("Loaded existing segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc());
+ LOGGER.info("Loaded existing segment: {} of table: {} with crc: {} on tier: {}", segmentName, _tableNameWithType,
+ zkMetadata.getCrc(), zkMetadata.getTier());
return true;
} catch (Exception e) {
- LOGGER.error("Failed to load existing segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc(), e);
+ LOGGER.error("Failed to load existing segment: {} of table: {} with crc: {} on tier: {}", segmentName,
+ _tableNameWithType, zkMetadata.getCrc(), zkMetadata.getTier(), e);
closeSegmentDirectoryQuietly(segmentDirectory);
return false;
}
}
- private SegmentDirectory tryInitSegmentDirectory(String segmentName, String segmentCrc,
+ private SegmentDirectory tryInitSegmentDirectory(String segmentName, String segmentCrc, String segmentTier,
IndexLoadingConfig indexLoadingConfig, Schema schema) {
try {
- return initSegmentDirectory(segmentName, segmentCrc, indexLoadingConfig, schema);
+ return initSegmentDirectory(segmentName, segmentCrc, segmentTier, indexLoadingConfig, schema);
} catch (Exception e) {
- LOGGER.warn("Failed to initialize SegmentDirectory for segment: {} of table: {} with error: {}", segmentName,
- _tableNameWithType, e.getMessage());
+ LOGGER.warn("Failed to initialize SegmentDirectory for segment: {} of table: {} on tier: {} with error: {}",
+ segmentName, _tableNameWithType, segmentTier, e.getMessage());
return null;
}
}
- private SegmentDirectory initSegmentDirectory(String segmentName, String segmentCrc,
+ private SegmentDirectory initSegmentDirectory(String segmentName, String segmentCrc, String segmentTier,
IndexLoadingConfig indexLoadingConfig, Schema schema)
throws Exception {
SegmentDirectoryLoaderContext loaderContext =
new SegmentDirectoryLoaderContext.Builder().setTableConfig(indexLoadingConfig.getTableConfig())
- .setSchema(schema).setInstanceId(indexLoadingConfig.getInstanceId()).setSegmentName(segmentName)
- .setSegmentCrc(segmentCrc).setSegmentDirectoryConfigs(indexLoadingConfig.getSegmentDirectoryConfigs())
- .build();
+ .setSchema(schema).setInstanceId(indexLoadingConfig.getInstanceId()).setTableDataDir(_tableDataDir)
+ .setSegmentName(segmentName).setSegmentCrc(segmentCrc).setSegmentTier(segmentTier)
+ .setSegmentDirectoryConfigs(indexLoadingConfig.getSegmentDirectoryConfigs()).build();
SegmentDirectoryLoader segmentDirectoryLoader =
SegmentDirectoryLoaderRegistry.getSegmentDirectoryLoader(indexLoadingConfig.getSegmentDirectoryLoader());
- File indexDir = getSegmentDataDir(segmentName);
+ File indexDir = getSegmentDataDir(segmentName, segmentTier, indexLoadingConfig);
return segmentDirectoryLoader.load(indexDir.toURI(), loaderContext);
Review Comment:
refactored a bit. The src/dest tiers are called out clearly in TierBasedSegmentDirectoryLoader as that's where tier migration actually happens. As this is a class common for all kinda of loader, I didn't expand too much on the tier concept here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on PR #9306:
URL: https://github.com/apache/pinot/pull/9306#issuecomment-1239942838
> What will happen if the tier is updated when server is restarting? Server won't be able to find the old local segment, and the old segment will left as orphan and never be cleaned up.
Good question. Although will add a cleanup mechanism for orphan segments as part of this work, it's still good to find and reuse the existing segment on disk under such race condition, to avoid orphan segment and downloading raw segment.
But as to the ideas `persist segment tier over server restarts` or `check all previous tiers`, they are subject to another race condition: the dataDir for tier is updated in TierConfig in TableConfig when server restarts. So instead, we may simply persist the segmentDir path over restarts, and check it out when calling addOrReplaceSegment(). This check is supposed to be done in TierBasedSegmentDirectoryLoader as those race conditions are specific to it. Will think a bit more on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r979075807
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -347,19 +349,21 @@ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingCon
}
indexDir = downloadSegment(segmentName, zkMetadata);
} else {
- LOGGER.info("Reload existing segment: {} of table: {}", segmentName, _tableNameWithType);
+ LOGGER.info("Reload existing segment: {} of table: {} currently on tier: {}", segmentName, _tableNameWithType,
Review Comment:
(minor) In most of the cases, tier will be `null`. Consider adding an if check instead of always logging the tier info
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -347,19 +349,21 @@ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingCon
}
indexDir = downloadSegment(segmentName, zkMetadata);
} else {
- LOGGER.info("Reload existing segment: {} of table: {}", segmentName, _tableNameWithType);
+ LOGGER.info("Reload existing segment: {} of table: {} currently on tier: {}", segmentName, _tableNameWithType,
+ segmentTier);
// The indexDir is empty after calling createBackup, as it's renamed to a backup directory.
// The SegmentDirectory should initialize accordingly. Like for SegmentLocalFSDirectory, it
// doesn't load anything from an empty indexDir, but gets the info to complete the copyTo.
try (SegmentDirectory segmentDirectory = initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()),
- indexLoadingConfig, schema)) {
+ segmentTier, indexLoadingConfig, schema)) {
segmentDirectory.copyTo(indexDir);
}
}
// Load from indexDir and replace the old segment in memory. What's inside indexDir
// may come from SegmentDirectory.copyTo() or the segment downloaded from deep store.
- ImmutableSegment segment = ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema);
+ ImmutableSegment segment =
Review Comment:
Consider wrapping `tableDataDir` and `tier` into the `indexLoadingConfig` to reduce the parameters passed.
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -329,7 +330,8 @@ public Map<String, SegmentErrorInfo> getSegmentErrors() {
public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig, SegmentZKMetadata zkMetadata,
SegmentMetadata localMetadata, @Nullable Schema schema, boolean forceDownload)
throws Exception {
- File indexDir = getSegmentDataDir(segmentName);
+ String segmentTier = getSegmentCurrentTier(segmentName);
Review Comment:
We should store the tier name within the `localMetadata`. You should be able to set it within the `ImmutableSegmentLoader.load(SegmentDirectory segmentDirectory, IndexLoadingConfig indexLoadingConfig, @Nullable Schema schema)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r995119924
##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java:
##########
@@ -226,6 +230,45 @@ public void removeSegment(String tableNameWithType, String segmentName) {
});
}
+ @Override
+ public void dropSegment(String tableNameWithType, String segmentName)
+ throws Exception {
+ // This method might modify the file on disk. Use segment lock to prevent race condition
+ Lock segmentLock = SegmentLocks.getSegmentLock(tableNameWithType, segmentName);
+ try {
+ segmentLock.lock();
+
+ // Clean up the segment data on default tier unconditionally.
+ File segmentDir = getSegmentDataDirectory(tableNameWithType, segmentName);
+ if (segmentDir.exists()) {
+ FileUtils.deleteQuietly(segmentDir);
+ LOGGER.info("Deleted segment directory {} on default tier", segmentDir);
+ }
+ // Note that this method usually happens after removeSegment, which removes the segment object from the
+ // tableDataManager object already, so we can't check segment info from there. In addition, tableDataManager
+ // object itself might not be present for the given table on the server either. So the locations of segment data
+ // are derived from configs and cached info at best effort.
+ TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
Review Comment:
right, current way would need to drop segments firstly before drop table config.
the `local tier file` is a detail of TierBasedSegmentDirectoryLoader, so didn't expose that here.
TierBasedSegmentDirectoryLoader would need TableConfig (the tier configs to be specific) to clean up the segment data from disk in last known tier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r965243931
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/config/TierConfigUtils.java:
##########
@@ -47,6 +51,27 @@ public static boolean shouldRelocateToTiers(TableConfig tableConfig) {
return CollectionUtils.isNotEmpty(tableConfig.getTierConfigsList());
}
+ public static String getDataDirFromTierConfig(String tableNameWithType, String tierName, TableConfig tableConfig) {
Review Comment:
(minor) We don't need to pass in table name because that can be read from the table config. Also suggest renaming the method:
```suggestion
public static String getDataDirForTier(TableConfig tableConfig, String tierName) {
```
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/config/TierConfigUtils.java:
##########
@@ -47,6 +51,27 @@ public static boolean shouldRelocateToTiers(TableConfig tableConfig) {
return CollectionUtils.isNotEmpty(tableConfig.getTierConfigsList());
}
+ public static String getDataDirFromTierConfig(String tableNameWithType, String tierName, TableConfig tableConfig) {
+ List<TierConfig> tierCfgs = tableConfig.getTierConfigsList();
+ Preconditions.checkState(CollectionUtils.isNotEmpty(tierCfgs), "No tierConfigs for table: %s", tableNameWithType);
+ TierConfig tierCfg = null;
+ for (TierConfig tc : tierCfgs) {
+ if (tierName.equalsIgnoreCase(tc.getName())) {
Review Comment:
(minor) Should we do exact match?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on PR #9306:
URL: https://github.com/apache/pinot/pull/9306#issuecomment-1239905005
> When addOrReplaceSegment() is called, as there is no segment loaded yet, the segmentZKMetadata.tier is used to load the segment for the first time on server. When reloadSegment() is called, as the segment is loaded by server (and we're reloading it), the current tier tracked by the ImmutableSegment object is compared with segmentZKMetadata.tier to decide if there is need to move indexDir to a new dir before reloading the segment.
What will happen if the tier is updated when server is restarting? Server won't be able to find the old local segment, and the old segment will left as orphan and never be cleaned up.
I think we need to find a way to persist the current tier info to the disk so that when server restarts, it can find the local segment tier. Or we may search the previous tiers if we cannot find the local segment, and migrate the tier immediately if the local segment is in the data dir for another tier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: MultiDirSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r964220968
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -406,9 +410,12 @@ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoad
localMetadata.getCrc(), zkMetadata.getCrc());
}
File indexDir = downloadSegment(segmentName, zkMetadata);
- addSegment(indexDir, indexLoadingConfig);
- LOGGER.info("Downloaded and loaded segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc());
+ Schema schema = ZKMetadataProvider.getTableSchema(_propertyStore, _tableNameWithType);
Review Comment:
mark it deprecated for now. I see it is only called in a method, which is not used any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #9306: MultiDirSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9306:
URL: https://github.com/apache/pinot/pull/9306#issuecomment-1233231267
# [Codecov](https://codecov.io/gh/apache/pinot/pull/9306?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9306](https://codecov.io/gh/apache/pinot/pull/9306?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (cafb4e3) into [master](https://codecov.io/gh/apache/pinot/commit/15b16e8f1029c745397dff869fd0759434761011?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (15b16e8) will **decrease** coverage by `43.64%`.
> The diff coverage is `21.42%`.
```diff
@@ Coverage Diff @@
## master #9306 +/- ##
=============================================
- Coverage 69.82% 26.18% -43.65%
+ Complexity 4771 44 -4727
=============================================
Files 1867 1862 -5
Lines 99625 99309 -316
Branches 15150 15131 -19
=============================================
- Hits 69568 26007 -43561
- Misses 25122 70684 +45562
+ Partials 4935 2618 -2317
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `26.18% <21.42%> (+0.03%)` | :arrow_up: |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che/pinot/common/utils/config/TierConfigUtils.java](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvY29uZmlnL1RpZXJDb25maWdVdGlscy5qYXZh) | `2.32% <0.00%> (-49.53%)` | :arrow_down: |
| [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-85.46%)` | :arrow_down: |
| [...t/local/loader/MultiDirSegmentDirectoryLoader.java](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9sb2FkZXIvTXVsdGlEaXJTZWdtZW50RGlyZWN0b3J5TG9hZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ment/spi/loader/SegmentDirectoryLoaderContext.java](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2xvYWRlci9TZWdtZW50RGlyZWN0b3J5TG9hZGVyQ29udGV4dC5qYXZh) | `0.00% <0.00%> (-82.15%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/CommonConstants.java](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQ29tbW9uQ29uc3RhbnRzLmphdmE=) | `0.00% <0.00%> (-27.70%)` | :arrow_down: |
| [...not/common/metadata/segment/SegmentZKMetadata.java](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0YWRhdGEvc2VnbWVudC9TZWdtZW50WktNZXRhZGF0YS5qYXZh) | `80.66% <33.33%> (-5.73%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `57.82% <58.97%> (-15.74%)` | :arrow_down: |
| [...in/java/org/apache/pinot/spi/utils/BytesUtils.java](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQnl0ZXNVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...java/org/apache/pinot/spi/trace/BaseRecording.java](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdHJhY2UvQmFzZVJlY29yZGluZy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1375 more](https://codecov.io/gh/apache/pinot/pull/9306/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on PR #9306:
URL: https://github.com/apache/pinot/pull/9306#issuecomment-1241099058
> But as to the ideas `persist segment tier over server restarts` or `check all previous tiers`, they are subject to another race condition: the dataDir for tier is updated in TierConfig in TableConfig when server restarts. So instead, we may simply persist the segmentDir path over restarts, and check it out when calling addOrReplaceSegment(). This check is supposed to be done in TierBasedSegmentDirectoryLoader as those race conditions are specific to it. Will think a bit more on this.
I thought about the problem of changing data dir, which is a destructive operation anyway (think of the case of server changing its root data dir). That leads me start thinking whether we should store the tier data dir in the table config. IMO that should be a server config because each server might have disk mounted under different path (not likely happen but just an example why it should be part of the server config). Basically the mapping from tier name to data dir should be within the server config, and the table config should only specify the tier name to use.
Persisting local tier requires more management overhead such as where to store this meta, when the update it, clean it up when segment is removed etc. That's why checking all previous tiers might be easier.
Segment deletion is another problem we need to think of. When segment is deleted, the ZK metadata is already gone. How do we find the local segment to remove?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r979113119
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -329,7 +330,8 @@ public Map<String, SegmentErrorInfo> getSegmentErrors() {
public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig, SegmentZKMetadata zkMetadata,
SegmentMetadata localMetadata, @Nullable Schema schema, boolean forceDownload)
throws Exception {
- File indexDir = getSegmentDataDir(segmentName);
+ String segmentTier = getSegmentCurrentTier(segmentName);
Review Comment:
Understood, but logically the tier info should be part of the metadata. E.g `indexDir` is not from the properties file. For the mutable one, it can simply return `null`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: MultiDirSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r963977609
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -546,6 +553,31 @@ File getSegmentDataDir(String segmentName) {
return new File(_indexDir, segmentName);
}
+ @VisibleForTesting
+ File getSegmentDataDir(String segmentName, String segmentTier, IndexLoadingConfig indexLoadingConfig) {
+ if (segmentTier == null) {
+ return getSegmentDataDir(segmentName);
+ }
+ try {
+ String tierDataDir = TierConfigUtils
+ .getDataDirFromTierConfig(_tableNameWithType, segmentTier, indexLoadingConfig.getTableConfig());
+ File tierTableDataDir = new File(tierDataDir, _tableNameWithType);
+ return new File(tierTableDataDir, segmentName);
+ } catch (Exception e) {
+ LOGGER.warn("Failed to get dataDir for segment: {} of table: {} on tier: {} due to error: {}", segmentName,
+ _tableNameWithType, segmentTier, e.getMessage());
+ return getSegmentDataDir(segmentName);
+ }
+ }
+
+ private String getSegmentCurrentTier(String segmentName) {
Review Comment:
kk, I thought those annotations are for public methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r995148988
##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java:
##########
@@ -226,6 +230,45 @@ public void removeSegment(String tableNameWithType, String segmentName) {
});
}
+ @Override
+ public void dropSegment(String tableNameWithType, String segmentName)
+ throws Exception {
+ // This method might modify the file on disk. Use segment lock to prevent race condition
+ Lock segmentLock = SegmentLocks.getSegmentLock(tableNameWithType, segmentName);
+ try {
+ segmentLock.lock();
+
+ // Clean up the segment data on default tier unconditionally.
+ File segmentDir = getSegmentDataDirectory(tableNameWithType, segmentName);
+ if (segmentDir.exists()) {
+ FileUtils.deleteQuietly(segmentDir);
+ LOGGER.info("Deleted segment directory {} on default tier", segmentDir);
+ }
+ // Note that this method usually happens after removeSegment, which removes the segment object from the
+ // tableDataManager object already, so we can't check segment info from there. In addition, tableDataManager
+ // object itself might not be present for the given table on the server either. So the locations of segment data
+ // are derived from configs and cached info at best effort.
+ TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
Review Comment:
as discussed, will keep both tier+path in the track file to `not` depend on any remote configs when cleaning up local states.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r995148514
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/InstanceDataManager.java:
##########
@@ -86,11 +86,17 @@ void addRealtimeSegment(String realtimeTableName, String segmentName)
throws Exception;
/**
- * Removes a segment from a table.
+ * Removes a segment from a table but not dropping its data from server.
*/
void removeSegment(String tableNameWithType, String segmentName)
throws Exception;
+ /**
+ * Delete segment data from the server.
+ */
+ void dropSegment(String tableNameWithType, String segmentName)
Review Comment:
will rename removeSegment to offloadSegment() and call this one deleteSegment()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged PR #9306:
URL: https://github.com/apache/pinot/pull/9306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: MultiDirSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r963980370
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -642,44 +675,44 @@ private boolean tryLoadExistingSegment(String segmentName, IndexLoadingConfig in
// Close the stale SegmentDirectory object and recreate it with reprocessed segment.
closeSegmentDirectoryQuietly(segmentDirectory);
ImmutableSegmentLoader.preprocess(indexDir, indexLoadingConfig, schema);
- segmentDirectory =
- initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()), indexLoadingConfig, schema);
+ segmentDirectory = initSegmentDirectory(segmentName, String.valueOf(zkMetadata.getCrc()), zkMetadata.getTier(),
+ indexLoadingConfig, schema);
}
ImmutableSegment segment = ImmutableSegmentLoader.load(segmentDirectory, indexLoadingConfig, schema);
addSegment(segment);
- LOGGER.info("Loaded existing segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc());
+ LOGGER.info("Loaded existing segment: {} of table: {} with crc: {} on tier: {}", segmentName, _tableNameWithType,
+ zkMetadata.getCrc(), zkMetadata.getTier());
return true;
} catch (Exception e) {
- LOGGER.error("Failed to load existing segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
- zkMetadata.getCrc(), e);
+ LOGGER.error("Failed to load existing segment: {} of table: {} with crc: {} on tier: {}", segmentName,
+ _tableNameWithType, zkMetadata.getCrc(), zkMetadata.getTier(), e);
closeSegmentDirectoryQuietly(segmentDirectory);
return false;
}
}
- private SegmentDirectory tryInitSegmentDirectory(String segmentName, String segmentCrc,
+ private SegmentDirectory tryInitSegmentDirectory(String segmentName, String segmentCrc, String segmentTier,
IndexLoadingConfig indexLoadingConfig, Schema schema) {
try {
- return initSegmentDirectory(segmentName, segmentCrc, indexLoadingConfig, schema);
+ return initSegmentDirectory(segmentName, segmentCrc, segmentTier, indexLoadingConfig, schema);
} catch (Exception e) {
- LOGGER.warn("Failed to initialize SegmentDirectory for segment: {} of table: {} with error: {}", segmentName,
- _tableNameWithType, e.getMessage());
+ LOGGER.warn("Failed to initialize SegmentDirectory for segment: {} of table: {} on tier: {} with error: {}",
+ segmentName, _tableNameWithType, segmentTier, e.getMessage());
return null;
}
}
- private SegmentDirectory initSegmentDirectory(String segmentName, String segmentCrc,
+ private SegmentDirectory initSegmentDirectory(String segmentName, String segmentCrc, String segmentTier,
IndexLoadingConfig indexLoadingConfig, Schema schema)
throws Exception {
SegmentDirectoryLoaderContext loaderContext =
new SegmentDirectoryLoaderContext.Builder().setTableConfig(indexLoadingConfig.getTableConfig())
- .setSchema(schema).setInstanceId(indexLoadingConfig.getInstanceId()).setSegmentName(segmentName)
- .setSegmentCrc(segmentCrc).setSegmentDirectoryConfigs(indexLoadingConfig.getSegmentDirectoryConfigs())
- .build();
+ .setSchema(schema).setInstanceId(indexLoadingConfig.getInstanceId()).setTableDataDir(_tableDataDir)
+ .setSegmentName(segmentName).setSegmentCrc(segmentCrc).setSegmentTier(segmentTier)
+ .setSegmentDirectoryConfigs(indexLoadingConfig.getSegmentDirectoryConfigs()).build();
SegmentDirectoryLoader segmentDirectoryLoader =
SegmentDirectoryLoaderRegistry.getSegmentDirectoryLoader(indexLoadingConfig.getSegmentDirectoryLoader());
- File indexDir = getSegmentDataDir(segmentName);
+ File indexDir = getSegmentDataDir(segmentName, segmentTier, indexLoadingConfig);
return segmentDirectoryLoader.load(indexDir.toURI(), loaderContext);
Review Comment:
the provided segmentTier is the `current tier` of the segment. This `initSegmentDirectory` is a util method to initialize a SegmentDirectory object for the segment from the current tier to complete the following logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a diff in pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on code in PR #9306:
URL: https://github.com/apache/pinot/pull/9306#discussion_r965290343
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/config/TierConfigUtils.java:
##########
@@ -47,6 +51,27 @@ public static boolean shouldRelocateToTiers(TableConfig tableConfig) {
return CollectionUtils.isNotEmpty(tableConfig.getTierConfigsList());
}
+ public static String getDataDirFromTierConfig(String tableNameWithType, String tierName, TableConfig tableConfig) {
+ List<TierConfig> tierCfgs = tableConfig.getTierConfigsList();
+ Preconditions.checkState(CollectionUtils.isNotEmpty(tierCfgs), "No tierConfigs for table: %s", tableNameWithType);
+ TierConfig tierCfg = null;
+ for (TierConfig tc : tierCfgs) {
+ if (tierName.equalsIgnoreCase(tc.getName())) {
Review Comment:
I saw insensitive case comparison used in a place. but on a double check, there are also Maps built with tier name directly. It seems simpler to always do case sensitive comparison on tier name. will update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on pull request #9306: TierBasedSegmentDirectoryLoader to keep segments in multi-datadir
Posted by GitBox <gi...@apache.org>.
klsince commented on PR #9306:
URL: https://github.com/apache/pinot/pull/9306#issuecomment-1239881053
> Where is the current tier stored?
It's set when immutable segment is loaded. TieredStorageLoader sets it after loading the segment.
```
ImmutableSegment.getTier() {
return _segmentDirectory.getTier();
}
TierBasedSegmentDirectoryLoader.load() {
...
segmentDirectory.setTier(segmentTier);
...
}
```
When addOrReplaceSegment() is called, as there is no segment loaded yet, the segmentZKMetadata.tier is used to load the segment for the first time on server. When reloadSegment() is called, as the segment is loaded by server (and we're reloading it), the current tier tracked by the ImmutableSegment object is compared with segmentZKMetadata.tier to decide if there is need to move indexDir to a new dir before reloading the segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org