You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/05/28 18:15:48 UTC
[GitHub] [hudi] bvaradar opened a new pull request #1678: [WIP] [HUDI-242] Metadata Bootstrap changes
bvaradar opened a new pull request #1678:
URL: https://github.com/apache/hudi/pull/1678
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar closed pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar closed pull request #1678:
URL: https://github.com/apache/hudi/pull/1678
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r463907314
##########
File path: hudi-client/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieKey;
+
+import org.apache.avro.generic.GenericRecord;
+
+import java.io.Serializable;
+import org.apache.hudi.exception.HoodieKeyException;
+
+/**
+ * Abstract class to extend for plugging in extraction of {@link HoodieKey} from an Avro record.
+ */
+public abstract class KeyGenerator implements Serializable {
+
+ protected transient TypedProperties config;
+
+ protected KeyGenerator(TypedProperties config) {
+ this.config = config;
+ }
+
+ /**
+ * Generate a record Key out of provided generic record.
+ */
+ public abstract String getRecordKey(GenericRecord record);
Review comment:
I created a `BuiltinKeyGenerator` and moved all our refactoring under there. external KeyGenerator need not change unless they want bootstrap support. if so, they have to implement one method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r442164224
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
Review comment:
this should live somewhere more central, in a enum somewhere the existing extensions live?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
Review comment:
we should probably rework this to write using the `HoodieTable` abstraction, based on the other PR.. but can happen in a follow on JIRA may be instead of fixing this one..
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
Review comment:
index being present and being empty are two separate things right? may be `isEmpty` or `isPresent` .. having a `hasIndex` inside a class named `..Index` is bit confusing.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
+ } else {
+ hasIndex = false;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ private static String getPartitionKey(String partition) {
+ return "part=" + partition;
+ }
+
+ private static String getFileGroupKey(HoodieFileGroupId fileGroupId) {
+ return "part=" + fileGroupId.getPartitionPath() + ";fileid=" + fileGroupId.getFileId();
+ }
+
+ private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
Review comment:
rename: getPartitionPathIndex
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -545,4 +556,52 @@ public static boolean isGCSInputStream(FSDataInputStream inputStream) {
|| inputStream.getWrappedStream().getClass().getCanonicalName()
.equals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream");
}
+
+ /**
+ * Get the FS implementation for this table.
+ * @param path Path String
+ * @param hadoopConf Serializable Hadoop Configuration
+ * @param consistencyGuardConfig Consistency Guard Config
+ * @return HoodieWrapperFileSystem
+ */
+ public static HoodieWrapperFileSystem getFs(String path, SerializableConfiguration hadoopConf,
+ ConsistencyGuardConfig consistencyGuardConfig) {
+ FileSystem fileSystem = FSUtils.getFs(path, hadoopConf.newCopy());
+ //Preconditions.checkArgument(!(fileSystem instanceof HoodieWrapperFileSystem),
+ // "File System not expected to be that of HoodieWrapperFileSystem");
+ return new HoodieWrapperFileSystem(fileSystem,
+ consistencyGuardConfig.isConsistencyCheckEnabled()
+ ? new FailSafeConsistencyGuard(fileSystem, consistencyGuardConfig)
+ : new NoOpConsistencyGuard());
+ }
+
+ public static List<Pair<String, List<HoodieFileStatus>>> getAllLeafFoldersWithFiles(FileSystem fs, String basePathStr,
Review comment:
lets make sure this has unit tests? also if this specific to bootstrap.. should this live here or in a `BootstrapUtils` or something.. we should look into redistributing all the code in this class and preferrably not pile on more..
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
+ } else {
+ hasIndex = false;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ private static String getPartitionKey(String partition) {
+ return "part=" + partition;
+ }
+
+ private static String getFileGroupKey(HoodieFileGroupId fileGroupId) {
+ return "part=" + fileGroupId.getPartitionPath() + ";fileid=" + fileGroupId.getFileId();
+ }
+
+ private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByPartitionFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ private static Path getIndexByFileIdPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByFileIdFolderNameFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ /**
+ * Helper method to create HFile Reader.
+ *
+ * @param hFilePath File Path
+ * @param conf Configuration
+ * @param fileSystem File System
+ */
+ private static HFile.Reader createReader(String hFilePath, Configuration conf, FileSystem fileSystem) {
+ try {
+ LOG.info("Opening HFile for reading :" + hFilePath);
+ HFile.Reader reader = HFile.createReader(fileSystem, new HFilePathForReader(hFilePath),
+ new CacheConfig(conf), conf);
+ return reader;
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public BootstrapIndex.IndexReader createReader() {
+ return new HFileBootstrapIndexReader(metaClient);
+ }
+
+ @Override
+ public BootstrapIndex.IndexWriter createWriter(String sourceBasePath) {
+ return new HFileBootstrapIndexWriter(sourceBasePath, metaClient);
+ }
+
+ @Override
+ public void dropIndex() {
+ try {
+ Path[] indexPaths = new Path[]{getIndexByPartitionPath(metaClient), getIndexByFileIdPath(metaClient)};
+ for (Path indexPath : indexPaths) {
+ if (metaClient.getFs().exists(indexPath)) {
+ LOG.info("Dropping bootstrap index. Deleting file : " + indexPath);
+ metaClient.getFs().delete(indexPath);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ protected boolean checkIndex() {
+ return hasIndex;
+ }
+
+ /**
+ * HFile Based Index Reader.
+ */
+ public static class HFileBootstrapIndexReader extends BootstrapIndex.IndexReader {
+
+ // Base Path of external files.
+ private final String sourceBasePath;
+ // Well Known Paths for indices
+ private final String indexByPartitionPath;
+ private final String indexByFileIdPath;
+
+ // Index Readers
+ private transient HFile.Reader indexByPartitionReader;
+ private transient HFile.Reader indexByFileIdReader;
+
+ // Bootstrap Index Info
+ private transient BootstrapIndexInfo bootstrapIndexInfo;
+
+ public HFileBootstrapIndexReader(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ this.indexByPartitionPath = indexByPartitionPath.toString();
+ this.indexByFileIdPath = indexByFilePath.toString();
+ this.sourceBasePath = getIndexInfo().getSourceBasePath();
+ LOG.info("Loaded HFileBasedBootstrapIndex with source base path :" + sourceBasePath);
+ }
+
+ @Override
+ public BootstrapIndexInfo getIndexInfo() {
+ if (null == bootstrapIndexInfo) {
+ synchronized (this) {
+ if (null == bootstrapIndexInfo) {
+ try {
+ bootstrapIndexInfo = fetchBootstrapIndexInfo();
+ } catch (IOException ioe) {
+ throw new HoodieException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+ }
+ return bootstrapIndexInfo;
+ }
+
+ private BootstrapIndexInfo fetchBootstrapIndexInfo() throws IOException {
+ return TimelineMetadataUtils.deserializeAvroMetadata(
+ getIndexByPartitionReader().loadFileInfo().get(INDEX_INFO_KEY),
+ BootstrapIndexInfo.class);
+ }
+
+ private HFile.Reader getIndexByPartitionReader() {
+ if (null == indexByPartitionReader) {
+ synchronized (this) {
+ if (null == indexByPartitionReader) {
+ LOG.info("Opening partition index :" + indexByPartitionPath);
+ this.indexByPartitionReader =
+ createReader(indexByPartitionPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByPartitionReader;
+ }
+
+ private HFile.Reader getIndexByFileIdReader() {
+ if (null == indexByFileIdReader) {
+ synchronized (this) {
+ if (null == indexByFileIdReader) {
+ LOG.info("Opening fileId index :" + indexByFileIdPath);
+ this.indexByFileIdReader =
+ createReader(indexByFileIdPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByFileIdReader;
+ }
+
+ @Override
+ public List<String> getIndexedPartitions() {
+ HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+ return getAllKeys(scanner);
+ }
+
+ @Override
+ public List<String> getIndexedFileIds() {
+ HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+ return getAllKeys(scanner);
+ }
+
+ private List<String> getAllKeys(HFileScanner scanner) {
+ List<String> keys = new ArrayList<>();
+ try {
+ boolean available = scanner.seekTo();
+ while (available) {
+ keys.add(CellUtil.getCellKeyAsString(scanner.getKeyValue()));
+ available = scanner.next();
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+
+ return keys;
+ }
+
+ @Override
+ public List<BootstrapSourceFileMapping> getSourceFileMappingForPartition(String partition) {
+ try {
+ HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+ KeyValue keyValue = new KeyValue(Bytes.toBytes(getPartitionKey(partition)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+ if (scanner.seekTo(keyValue) == 0) {
+ ByteBuffer readValue = scanner.getValue();
+ byte[] valBytes = Bytes.toBytes(readValue);
+ BootstrapPartitionMetadata metadata =
+ TimelineMetadataUtils.deserializeAvroMetadata(valBytes, BootstrapPartitionMetadata.class);
+ return metadata.getHudiFileIdToSourceFile().entrySet().stream()
+ .map(e -> new BootstrapSourceFileMapping(sourceBasePath, metadata.getSourcePartitionPath(),
+ partition, e.getValue(), e.getKey())).collect(Collectors.toList());
+ } else {
+ LOG.info("No value found for partition key (" + partition + ")");
+ return new ArrayList<>();
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public String getSourceBasePath() {
+ return sourceBasePath;
+ }
+
+ @Override
+ public Map<HoodieFileGroupId, BootstrapSourceFileMapping> getSourceFileMappingForFileIds(
+ List<HoodieFileGroupId> ids) {
+ Map<HoodieFileGroupId, BootstrapSourceFileMapping> result = new HashMap<>();
+ // Arrange input Keys in sorted order for 1 pass scan
+ List<HoodieFileGroupId> fileGroupIds = new ArrayList<>(ids);
+ Collections.sort(fileGroupIds);
+ try {
+ HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+ for (HoodieFileGroupId fileGroupId : fileGroupIds) {
+ KeyValue keyValue = new KeyValue(Bytes.toBytes(getFileGroupKey(fileGroupId)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+ if (scanner.seekTo(keyValue) == 0) {
+ ByteBuffer readValue = scanner.getValue();
+ byte[] valBytes = Bytes.toBytes(readValue);
+ BootstrapSourceFilePartitionInfo fileInfo = TimelineMetadataUtils.deserializeAvroMetadata(valBytes,
+ BootstrapSourceFilePartitionInfo.class);
+ BootstrapSourceFileMapping mapping = new BootstrapSourceFileMapping(sourceBasePath,
+ fileInfo.getSourcePartitionPath(), fileInfo.getHudiPartitionPath(), fileInfo.getSourceFileStatus(),
+ fileGroupId.getFileId());
+ result.put(fileGroupId, mapping);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ return result;
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (indexByPartitionReader != null) {
+ indexByPartitionReader.close(true);
+ indexByPartitionReader = null;
+ }
+ if (indexByFileIdReader != null) {
+ indexByFileIdReader.close(true);
+ indexByFileIdReader = null;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+
+ /**
+ * Boostrap Index Writer to build bootstrap index.
+ */
+ public static class HFileBootstrapIndexWriter extends BootstrapIndex.IndexWriter {
+
+ private final String sourceBasePath;
+ private final Path indexByPartitionPath;
+ private final Path indexByFileIdPath;
+ private HFile.Writer indexByPartitionWriter;
+ private HFile.Writer indexByFileIdWriter;
+
+ private boolean closed = false;
+ private int numPartitionKeysAdded = 0;
+ private int numFileIdKeysAdded = 0;
+
+ private final Map<String, List<BootstrapSourceFileMapping>> sourceFileMappings = new HashMap<>();
+
+ private HFileBootstrapIndexWriter(String sourceBasePath, HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ try {
+ metaClient.initializeBootstrapDirsIfNotExists();
+ this.sourceBasePath = sourceBasePath;
+ this.indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ this.indexByFileIdPath = getIndexByFileIdPath(metaClient);
+
+ if (metaClient.getFs().exists(indexByPartitionPath) || metaClient.getFs().exists(indexByFileIdPath)) {
+ String errMsg = "Previous version of bootstrap index exists. Partition Index Path :" + indexByPartitionPath
+ + ", FileId index Path :" + indexByFileIdPath;
+ LOG.info(errMsg);
+ throw new HoodieException(errMsg);
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ /**
+ * Append bootstrap index entries for next partitions in sorted order.
+ * @param hudiPartitionPath Hudi Partition Path
+ * @param sourcePartitionPath Source Partition Path
+ * @param bootstrapSourceFileMappings Bootstrap Source File to Hudi File Id mapping
+ */
+ private void writeNextPartition(String hudiPartitionPath, String sourcePartitionPath,
+ List<BootstrapSourceFileMapping> bootstrapSourceFileMappings) {
+ try {
+ LOG.info("Adding bootstrap partition Index entry for partition :" + hudiPartitionPath
+ + ", Source Partition :" + sourcePartitionPath + ", Num Entries :" + bootstrapSourceFileMappings.size());
+ LOG.info("ADDING entries :" + bootstrapSourceFileMappings);
+ BootstrapPartitionMetadata bootstrapPartitionMetadata = new BootstrapPartitionMetadata();
+ bootstrapPartitionMetadata.setSourcePartitionPath(sourcePartitionPath);
+ bootstrapPartitionMetadata.setHudiPartitionPath(hudiPartitionPath);
+ bootstrapPartitionMetadata.setHudiFileIdToSourceFile(
+ bootstrapSourceFileMappings.stream().map(m -> Pair.of(m.getHudiFileId(),
+ m.getSourceFileStatus())).collect(Collectors.toMap(Pair::getKey, Pair::getValue)));
+ Option<byte[]> bytes = TimelineMetadataUtils.serializeAvroMetadata(bootstrapPartitionMetadata,
+ BootstrapPartitionMetadata.class);
+ if (bytes.isPresent()) {
+ indexByPartitionWriter
+ .append(new KeyValue(Bytes.toBytes(getPartitionKey(hudiPartitionPath)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, bytes.get()));
+ numPartitionKeysAdded++;
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Write next source file to hudi file-id. Entries are expected to be appended in hudi file-group id
Review comment:
arghhh.. same :)
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
##########
@@ -312,6 +332,41 @@ protected FileSlice filterBaseFileAfterPendingCompaction(FileSlice fileSlice) {
return fileSlice;
}
+ protected HoodieFileGroup addExternalBaseFileIfPresent(HoodieFileGroup fileGroup) {
+ boolean hasExternalBaseFile = fileGroup.getAllFileSlices()
Review comment:
on a related note : we can do the check for bootstrap index isAvailable() after we determine the value of `hasExternalBaseFile`?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -302,22 +322,23 @@ public synchronized HoodieArchivedTimeline getArchivedTimeline() {
* Helper method to initialize a table, with given basePath, tableType, name, archiveFolder.
*/
public static HoodieTableMetaClient initTableType(Configuration hadoopConf, String basePath, String tableType,
- String tableName, String archiveLogFolder, String payloadClassName) throws IOException {
+ String tableName, String archiveLogFolder, String payloadClassName, String bootstrapIndexClass)
Review comment:
should it be Option<bootstrapIndexClass>?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -252,16 +272,24 @@ private static void cleanProjectionColumnIds(Configuration conf) {
LOG.info("Creating record reader with readCols :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR)
+ ", Ids :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR));
- // sanity check
- ValidationUtils.checkArgument(split instanceof HoodieRealtimeFileSplit,
- "HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit and not with " + split);
- return new HoodieRealtimeRecordReader((HoodieRealtimeFileSplit) split, jobConf,
- super.getRecordReader(split, jobConf, reporter));
+ // sanity check
+ ValidationUtils.checkArgument(split instanceof RealtimeSplit,
+ "HoodieRealtimeRecordReader can only work on RealtimeSplit and not with " + split);
+ JobConf jobConf1 = new JobConf(jobConf);
Review comment:
why the two copies? are they modified internally?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -607,4 +652,126 @@ private HoodieInstant fetchNextCompactionInstant() throws InterruptedException {
}, executor)).toArray(CompletableFuture[]::new)), executor);
}
}
+
+ /**
+ * Performs bootstrap from a non-hudi source.
+ */
+ public static class BootstrapExecutor implements Serializable {
Review comment:
pull into its own class ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
+ } else {
+ hasIndex = false;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ private static String getPartitionKey(String partition) {
+ return "part=" + partition;
+ }
+
+ private static String getFileGroupKey(HoodieFileGroupId fileGroupId) {
+ return "part=" + fileGroupId.getPartitionPath() + ";fileid=" + fileGroupId.getFileId();
+ }
+
+ private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByPartitionFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ private static Path getIndexByFileIdPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByFileIdFolderNameFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ /**
+ * Helper method to create HFile Reader.
+ *
+ * @param hFilePath File Path
+ * @param conf Configuration
+ * @param fileSystem File System
+ */
+ private static HFile.Reader createReader(String hFilePath, Configuration conf, FileSystem fileSystem) {
+ try {
+ LOG.info("Opening HFile for reading :" + hFilePath);
+ HFile.Reader reader = HFile.createReader(fileSystem, new HFilePathForReader(hFilePath),
+ new CacheConfig(conf), conf);
+ return reader;
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public BootstrapIndex.IndexReader createReader() {
+ return new HFileBootstrapIndexReader(metaClient);
+ }
+
+ @Override
+ public BootstrapIndex.IndexWriter createWriter(String sourceBasePath) {
+ return new HFileBootstrapIndexWriter(sourceBasePath, metaClient);
+ }
+
+ @Override
+ public void dropIndex() {
+ try {
+ Path[] indexPaths = new Path[]{getIndexByPartitionPath(metaClient), getIndexByFileIdPath(metaClient)};
+ for (Path indexPath : indexPaths) {
+ if (metaClient.getFs().exists(indexPath)) {
+ LOG.info("Dropping bootstrap index. Deleting file : " + indexPath);
+ metaClient.getFs().delete(indexPath);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ protected boolean checkIndex() {
+ return hasIndex;
+ }
+
+ /**
+ * HFile Based Index Reader.
+ */
+ public static class HFileBootstrapIndexReader extends BootstrapIndex.IndexReader {
+
+ // Base Path of external files.
+ private final String sourceBasePath;
+ // Well Known Paths for indices
+ private final String indexByPartitionPath;
+ private final String indexByFileIdPath;
+
+ // Index Readers
+ private transient HFile.Reader indexByPartitionReader;
+ private transient HFile.Reader indexByFileIdReader;
+
+ // Bootstrap Index Info
+ private transient BootstrapIndexInfo bootstrapIndexInfo;
+
+ public HFileBootstrapIndexReader(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ this.indexByPartitionPath = indexByPartitionPath.toString();
+ this.indexByFileIdPath = indexByFilePath.toString();
+ this.sourceBasePath = getIndexInfo().getSourceBasePath();
+ LOG.info("Loaded HFileBasedBootstrapIndex with source base path :" + sourceBasePath);
+ }
+
+ @Override
+ public BootstrapIndexInfo getIndexInfo() {
+ if (null == bootstrapIndexInfo) {
+ synchronized (this) {
+ if (null == bootstrapIndexInfo) {
+ try {
+ bootstrapIndexInfo = fetchBootstrapIndexInfo();
+ } catch (IOException ioe) {
+ throw new HoodieException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+ }
+ return bootstrapIndexInfo;
+ }
+
+ private BootstrapIndexInfo fetchBootstrapIndexInfo() throws IOException {
+ return TimelineMetadataUtils.deserializeAvroMetadata(
+ getIndexByPartitionReader().loadFileInfo().get(INDEX_INFO_KEY),
+ BootstrapIndexInfo.class);
+ }
+
+ private HFile.Reader getIndexByPartitionReader() {
+ if (null == indexByPartitionReader) {
+ synchronized (this) {
+ if (null == indexByPartitionReader) {
+ LOG.info("Opening partition index :" + indexByPartitionPath);
+ this.indexByPartitionReader =
+ createReader(indexByPartitionPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByPartitionReader;
+ }
+
+ private HFile.Reader getIndexByFileIdReader() {
+ if (null == indexByFileIdReader) {
+ synchronized (this) {
+ if (null == indexByFileIdReader) {
+ LOG.info("Opening fileId index :" + indexByFileIdPath);
+ this.indexByFileIdReader =
+ createReader(indexByFileIdPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByFileIdReader;
+ }
+
+ @Override
+ public List<String> getIndexedPartitions() {
Review comment:
consistent use of partition paths
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
+ } else {
+ hasIndex = false;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ private static String getPartitionKey(String partition) {
+ return "part=" + partition;
+ }
+
+ private static String getFileGroupKey(HoodieFileGroupId fileGroupId) {
+ return "part=" + fileGroupId.getPartitionPath() + ";fileid=" + fileGroupId.getFileId();
+ }
+
+ private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByPartitionFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ private static Path getIndexByFileIdPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByFileIdFolderNameFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ /**
+ * Helper method to create HFile Reader.
+ *
+ * @param hFilePath File Path
+ * @param conf Configuration
+ * @param fileSystem File System
+ */
+ private static HFile.Reader createReader(String hFilePath, Configuration conf, FileSystem fileSystem) {
Review comment:
let's separate helpers into another Utils class?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/InputSplitUtils.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import org.apache.avro.Schema;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+
+public class InputSplitUtils {
+
+ public static void writeString(String str, DataOutput out) throws IOException {
Review comment:
I thik I saw similar looking helpers some place else?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -63,11 +64,14 @@
public static final String HOODIE_TIMELINE_LAYOUT_VERSION = "hoodie.timeline.layout.version";
public static final String HOODIE_PAYLOAD_CLASS_PROP_NAME = "hoodie.compaction.payload.class";
public static final String HOODIE_ARCHIVELOG_FOLDER_PROP_NAME = "hoodie.archivelog.folder";
+ public static final String HOODIE_BOOTSTRAP_INDEX_CLASS_PROP_NAME = "hoodie.bootstrap.index.class";
public static final HoodieTableType DEFAULT_TABLE_TYPE = HoodieTableType.COPY_ON_WRITE;
public static final HoodieFileFormat DEFAULT_BASE_FILE_FORMAT = HoodieFileFormat.PARQUET;
public static final HoodieFileFormat DEFAULT_LOG_FILE_FORMAT = HoodieFileFormat.HOODIE_LOG;
public static final String DEFAULT_PAYLOAD_CLASS = OverwriteWithLatestAvroPayload.class.getName();
+ public static final String DEFAULT_BOOTSTRAP_INDEX_CLASS = HFileBasedBootstrapIndex.class.getName();
Review comment:
bootstrap base path can be written here too right?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
Review comment:
single line assignment?
isEmpty = a && b?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
+ } else {
+ hasIndex = false;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ private static String getPartitionKey(String partition) {
+ return "part=" + partition;
+ }
+
+ private static String getFileGroupKey(HoodieFileGroupId fileGroupId) {
+ return "part=" + fileGroupId.getPartitionPath() + ";fileid=" + fileGroupId.getFileId();
+ }
+
+ private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByPartitionFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ private static Path getIndexByFileIdPath(HoodieTableMetaClient metaClient) {
Review comment:
rename: getFileIdIndex
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
+ } else {
+ hasIndex = false;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ private static String getPartitionKey(String partition) {
+ return "part=" + partition;
+ }
+
+ private static String getFileGroupKey(HoodieFileGroupId fileGroupId) {
+ return "part=" + fileGroupId.getPartitionPath() + ";fileid=" + fileGroupId.getFileId();
+ }
+
+ private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByPartitionFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ private static Path getIndexByFileIdPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByFileIdFolderNameFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ /**
+ * Helper method to create HFile Reader.
+ *
+ * @param hFilePath File Path
+ * @param conf Configuration
+ * @param fileSystem File System
+ */
+ private static HFile.Reader createReader(String hFilePath, Configuration conf, FileSystem fileSystem) {
+ try {
+ LOG.info("Opening HFile for reading :" + hFilePath);
+ HFile.Reader reader = HFile.createReader(fileSystem, new HFilePathForReader(hFilePath),
+ new CacheConfig(conf), conf);
+ return reader;
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public BootstrapIndex.IndexReader createReader() {
+ return new HFileBootstrapIndexReader(metaClient);
+ }
+
+ @Override
+ public BootstrapIndex.IndexWriter createWriter(String sourceBasePath) {
+ return new HFileBootstrapIndexWriter(sourceBasePath, metaClient);
+ }
+
+ @Override
+ public void dropIndex() {
+ try {
+ Path[] indexPaths = new Path[]{getIndexByPartitionPath(metaClient), getIndexByFileIdPath(metaClient)};
+ for (Path indexPath : indexPaths) {
+ if (metaClient.getFs().exists(indexPath)) {
+ LOG.info("Dropping bootstrap index. Deleting file : " + indexPath);
+ metaClient.getFs().delete(indexPath);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ protected boolean checkIndex() {
+ return hasIndex;
+ }
+
+ /**
+ * HFile Based Index Reader.
+ */
+ public static class HFileBootstrapIndexReader extends BootstrapIndex.IndexReader {
+
+ // Base Path of external files.
+ private final String sourceBasePath;
+ // Well Known Paths for indices
+ private final String indexByPartitionPath;
+ private final String indexByFileIdPath;
+
+ // Index Readers
+ private transient HFile.Reader indexByPartitionReader;
+ private transient HFile.Reader indexByFileIdReader;
+
+ // Bootstrap Index Info
+ private transient BootstrapIndexInfo bootstrapIndexInfo;
+
+ public HFileBootstrapIndexReader(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ this.indexByPartitionPath = indexByPartitionPath.toString();
+ this.indexByFileIdPath = indexByFilePath.toString();
+ this.sourceBasePath = getIndexInfo().getSourceBasePath();
+ LOG.info("Loaded HFileBasedBootstrapIndex with source base path :" + sourceBasePath);
+ }
+
+ @Override
+ public BootstrapIndexInfo getIndexInfo() {
+ if (null == bootstrapIndexInfo) {
+ synchronized (this) {
+ if (null == bootstrapIndexInfo) {
+ try {
+ bootstrapIndexInfo = fetchBootstrapIndexInfo();
+ } catch (IOException ioe) {
+ throw new HoodieException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+ }
+ return bootstrapIndexInfo;
+ }
+
+ private BootstrapIndexInfo fetchBootstrapIndexInfo() throws IOException {
+ return TimelineMetadataUtils.deserializeAvroMetadata(
+ getIndexByPartitionReader().loadFileInfo().get(INDEX_INFO_KEY),
+ BootstrapIndexInfo.class);
+ }
+
+ private HFile.Reader getIndexByPartitionReader() {
+ if (null == indexByPartitionReader) {
+ synchronized (this) {
+ if (null == indexByPartitionReader) {
+ LOG.info("Opening partition index :" + indexByPartitionPath);
+ this.indexByPartitionReader =
+ createReader(indexByPartitionPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByPartitionReader;
+ }
+
+ private HFile.Reader getIndexByFileIdReader() {
+ if (null == indexByFileIdReader) {
Review comment:
is this optimization necessary? can we use an atomic reference or something if you are worried about concurrency
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/ExternalBaseFileMapping.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import java.io.Serializable;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+
+/**
+ * POJO storing (partitionPath, hoodieFileId) -> external data file path.
+ */
+public class ExternalBaseFileMapping implements Serializable {
Review comment:
you can hold any id, status mapping here right.. "External" .. for all we know this can be implemented usign a Pair class right?
let's make it generic `FileIdAndStatus` or something.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/BootstrapSourceFileMapping.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+
+/**
+ * Payload returned by Bootstrap Index.
Review comment:
this comment is very opaque.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
+ } else {
+ hasIndex = false;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ private static String getPartitionKey(String partition) {
+ return "part=" + partition;
+ }
+
+ private static String getFileGroupKey(HoodieFileGroupId fileGroupId) {
+ return "part=" + fileGroupId.getPartitionPath() + ";fileid=" + fileGroupId.getFileId();
+ }
+
+ private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByPartitionFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ private static Path getIndexByFileIdPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByFileIdFolderNameFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ /**
+ * Helper method to create HFile Reader.
+ *
+ * @param hFilePath File Path
+ * @param conf Configuration
+ * @param fileSystem File System
+ */
+ private static HFile.Reader createReader(String hFilePath, Configuration conf, FileSystem fileSystem) {
+ try {
+ LOG.info("Opening HFile for reading :" + hFilePath);
+ HFile.Reader reader = HFile.createReader(fileSystem, new HFilePathForReader(hFilePath),
+ new CacheConfig(conf), conf);
+ return reader;
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public BootstrapIndex.IndexReader createReader() {
+ return new HFileBootstrapIndexReader(metaClient);
+ }
+
+ @Override
+ public BootstrapIndex.IndexWriter createWriter(String sourceBasePath) {
+ return new HFileBootstrapIndexWriter(sourceBasePath, metaClient);
+ }
+
+ @Override
+ public void dropIndex() {
+ try {
+ Path[] indexPaths = new Path[]{getIndexByPartitionPath(metaClient), getIndexByFileIdPath(metaClient)};
+ for (Path indexPath : indexPaths) {
+ if (metaClient.getFs().exists(indexPath)) {
+ LOG.info("Dropping bootstrap index. Deleting file : " + indexPath);
+ metaClient.getFs().delete(indexPath);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ protected boolean checkIndex() {
+ return hasIndex;
+ }
+
+ /**
+ * HFile Based Index Reader.
+ */
+ public static class HFileBootstrapIndexReader extends BootstrapIndex.IndexReader {
+
+ // Base Path of external files.
+ private final String sourceBasePath;
+ // Well Known Paths for indices
+ private final String indexByPartitionPath;
+ private final String indexByFileIdPath;
+
+ // Index Readers
+ private transient HFile.Reader indexByPartitionReader;
+ private transient HFile.Reader indexByFileIdReader;
+
+ // Bootstrap Index Info
+ private transient BootstrapIndexInfo bootstrapIndexInfo;
+
+ public HFileBootstrapIndexReader(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ this.indexByPartitionPath = indexByPartitionPath.toString();
+ this.indexByFileIdPath = indexByFilePath.toString();
+ this.sourceBasePath = getIndexInfo().getSourceBasePath();
+ LOG.info("Loaded HFileBasedBootstrapIndex with source base path :" + sourceBasePath);
+ }
+
+ @Override
+ public BootstrapIndexInfo getIndexInfo() {
+ if (null == bootstrapIndexInfo) {
+ synchronized (this) {
+ if (null == bootstrapIndexInfo) {
+ try {
+ bootstrapIndexInfo = fetchBootstrapIndexInfo();
+ } catch (IOException ioe) {
+ throw new HoodieException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+ }
+ return bootstrapIndexInfo;
+ }
+
+ private BootstrapIndexInfo fetchBootstrapIndexInfo() throws IOException {
+ return TimelineMetadataUtils.deserializeAvroMetadata(
+ getIndexByPartitionReader().loadFileInfo().get(INDEX_INFO_KEY),
+ BootstrapIndexInfo.class);
+ }
+
+ private HFile.Reader getIndexByPartitionReader() {
+ if (null == indexByPartitionReader) {
+ synchronized (this) {
+ if (null == indexByPartitionReader) {
+ LOG.info("Opening partition index :" + indexByPartitionPath);
+ this.indexByPartitionReader =
+ createReader(indexByPartitionPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByPartitionReader;
+ }
+
+ private HFile.Reader getIndexByFileIdReader() {
+ if (null == indexByFileIdReader) {
+ synchronized (this) {
+ if (null == indexByFileIdReader) {
+ LOG.info("Opening fileId index :" + indexByFileIdPath);
+ this.indexByFileIdReader =
+ createReader(indexByFileIdPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByFileIdReader;
+ }
+
+ @Override
+ public List<String> getIndexedPartitions() {
+ HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+ return getAllKeys(scanner);
+ }
+
+ @Override
+ public List<String> getIndexedFileIds() {
+ HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+ return getAllKeys(scanner);
+ }
+
+ private List<String> getAllKeys(HFileScanner scanner) {
+ List<String> keys = new ArrayList<>();
+ try {
+ boolean available = scanner.seekTo();
+ while (available) {
+ keys.add(CellUtil.getCellKeyAsString(scanner.getKeyValue()));
+ available = scanner.next();
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+
+ return keys;
+ }
+
+ @Override
+ public List<BootstrapSourceFileMapping> getSourceFileMappingForPartition(String partition) {
+ try {
+ HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+ KeyValue keyValue = new KeyValue(Bytes.toBytes(getPartitionKey(partition)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+ if (scanner.seekTo(keyValue) == 0) {
+ ByteBuffer readValue = scanner.getValue();
+ byte[] valBytes = Bytes.toBytes(readValue);
+ BootstrapPartitionMetadata metadata =
+ TimelineMetadataUtils.deserializeAvroMetadata(valBytes, BootstrapPartitionMetadata.class);
+ return metadata.getHudiFileIdToSourceFile().entrySet().stream()
+ .map(e -> new BootstrapSourceFileMapping(sourceBasePath, metadata.getSourcePartitionPath(),
+ partition, e.getValue(), e.getKey())).collect(Collectors.toList());
+ } else {
+ LOG.info("No value found for partition key (" + partition + ")");
+ return new ArrayList<>();
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public String getSourceBasePath() {
+ return sourceBasePath;
+ }
+
+ @Override
+ public Map<HoodieFileGroupId, BootstrapSourceFileMapping> getSourceFileMappingForFileIds(
+ List<HoodieFileGroupId> ids) {
+ Map<HoodieFileGroupId, BootstrapSourceFileMapping> result = new HashMap<>();
+ // Arrange input Keys in sorted order for 1 pass scan
+ List<HoodieFileGroupId> fileGroupIds = new ArrayList<>(ids);
+ Collections.sort(fileGroupIds);
+ try {
+ HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+ for (HoodieFileGroupId fileGroupId : fileGroupIds) {
+ KeyValue keyValue = new KeyValue(Bytes.toBytes(getFileGroupKey(fileGroupId)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+ if (scanner.seekTo(keyValue) == 0) {
+ ByteBuffer readValue = scanner.getValue();
+ byte[] valBytes = Bytes.toBytes(readValue);
+ BootstrapSourceFilePartitionInfo fileInfo = TimelineMetadataUtils.deserializeAvroMetadata(valBytes,
+ BootstrapSourceFilePartitionInfo.class);
+ BootstrapSourceFileMapping mapping = new BootstrapSourceFileMapping(sourceBasePath,
+ fileInfo.getSourcePartitionPath(), fileInfo.getHudiPartitionPath(), fileInfo.getSourceFileStatus(),
+ fileGroupId.getFileId());
+ result.put(fileGroupId, mapping);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ return result;
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (indexByPartitionReader != null) {
+ indexByPartitionReader.close(true);
+ indexByPartitionReader = null;
+ }
+ if (indexByFileIdReader != null) {
+ indexByFileIdReader.close(true);
+ indexByFileIdReader = null;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+
+ /**
+ * Boostrap Index Writer to build bootstrap index.
+ */
+ public static class HFileBootstrapIndexWriter extends BootstrapIndex.IndexWriter {
+
+ private final String sourceBasePath;
+ private final Path indexByPartitionPath;
+ private final Path indexByFileIdPath;
+ private HFile.Writer indexByPartitionWriter;
+ private HFile.Writer indexByFileIdWriter;
+
+ private boolean closed = false;
+ private int numPartitionKeysAdded = 0;
+ private int numFileIdKeysAdded = 0;
+
+ private final Map<String, List<BootstrapSourceFileMapping>> sourceFileMappings = new HashMap<>();
+
+ private HFileBootstrapIndexWriter(String sourceBasePath, HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ try {
+ metaClient.initializeBootstrapDirsIfNotExists();
+ this.sourceBasePath = sourceBasePath;
+ this.indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ this.indexByFileIdPath = getIndexByFileIdPath(metaClient);
+
+ if (metaClient.getFs().exists(indexByPartitionPath) || metaClient.getFs().exists(indexByFileIdPath)) {
+ String errMsg = "Previous version of bootstrap index exists. Partition Index Path :" + indexByPartitionPath
+ + ", FileId index Path :" + indexByFileIdPath;
+ LOG.info(errMsg);
+ throw new HoodieException(errMsg);
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ /**
+ * Append bootstrap index entries for next partitions in sorted order.
+ * @param hudiPartitionPath Hudi Partition Path
+ * @param sourcePartitionPath Source Partition Path
+ * @param bootstrapSourceFileMappings Bootstrap Source File to Hudi File Id mapping
+ */
+ private void writeNextPartition(String hudiPartitionPath, String sourcePartitionPath,
Review comment:
naming stuff `hudi`
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -545,4 +556,52 @@ public static boolean isGCSInputStream(FSDataInputStream inputStream) {
|| inputStream.getWrappedStream().getClass().getCanonicalName()
.equals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream");
}
+
+ /**
+ * Get the FS implementation for this table.
+ * @param path Path String
+ * @param hadoopConf Serializable Hadoop Configuration
+ * @param consistencyGuardConfig Consistency Guard Config
+ * @return HoodieWrapperFileSystem
+ */
+ public static HoodieWrapperFileSystem getFs(String path, SerializableConfiguration hadoopConf,
+ ConsistencyGuardConfig consistencyGuardConfig) {
+ FileSystem fileSystem = FSUtils.getFs(path, hadoopConf.newCopy());
+ //Preconditions.checkArgument(!(fileSystem instanceof HoodieWrapperFileSystem),
Review comment:
remove?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
##########
@@ -368,9 +425,9 @@ protected FileSlice filterBaseFileAfterPendingCompaction(FileSlice fileSlice) {
String partitionPath = formatPartitionKey(partitionStr);
ensurePartitionLoadedCorrectly(partitionPath);
return fetchHoodieFileGroup(partitionPath, fileId).map(fileGroup -> fileGroup.getAllBaseFiles()
- .filter(
- baseFile -> HoodieTimeline.compareTimestamps(baseFile.getCommitTime(), HoodieTimeline.EQUALS, instantTime))
- .filter(df -> !isBaseFileDueToPendingCompaction(df)).findFirst().orElse(null));
+ .filter(baseFile -> HoodieTimeline.compareTimestamps(baseFile.getCommitTime(), HoodieTimeline.EQUALS,
Review comment:
more readable line folding?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -545,4 +556,52 @@ public static boolean isGCSInputStream(FSDataInputStream inputStream) {
|| inputStream.getWrappedStream().getClass().getCanonicalName()
.equals("com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream");
}
+
+ /**
+ * Get the FS implementation for this table.
+ * @param path Path String
+ * @param hadoopConf Serializable Hadoop Configuration
+ * @param consistencyGuardConfig Consistency Guard Config
+ * @return HoodieWrapperFileSystem
+ */
+ public static HoodieWrapperFileSystem getFs(String path, SerializableConfiguration hadoopConf,
Review comment:
rename:getWrapperFS()
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/BootstrapSourceFileMapping.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+
+/**
+ * Payload returned by Bootstrap Index.
+ */
+public class BootstrapSourceFileMapping implements Serializable, Comparable<BootstrapSourceFileMapping> {
+
+ private final String sourceBasePath;
Review comment:
why does each mapping need to store the `sourceBasePath` won't it be written to `hoodie.properties` and well known to any reading/writing?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
+ } else {
+ hasIndex = false;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ private static String getPartitionKey(String partition) {
+ return "part=" + partition;
+ }
+
+ private static String getFileGroupKey(HoodieFileGroupId fileGroupId) {
+ return "part=" + fileGroupId.getPartitionPath() + ";fileid=" + fileGroupId.getFileId();
+ }
+
+ private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByPartitionFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ private static Path getIndexByFileIdPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByFileIdFolderNameFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ /**
+ * Helper method to create HFile Reader.
+ *
+ * @param hFilePath File Path
+ * @param conf Configuration
+ * @param fileSystem File System
+ */
+ private static HFile.Reader createReader(String hFilePath, Configuration conf, FileSystem fileSystem) {
+ try {
+ LOG.info("Opening HFile for reading :" + hFilePath);
+ HFile.Reader reader = HFile.createReader(fileSystem, new HFilePathForReader(hFilePath),
+ new CacheConfig(conf), conf);
+ return reader;
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public BootstrapIndex.IndexReader createReader() {
+ return new HFileBootstrapIndexReader(metaClient);
+ }
+
+ @Override
+ public BootstrapIndex.IndexWriter createWriter(String sourceBasePath) {
+ return new HFileBootstrapIndexWriter(sourceBasePath, metaClient);
+ }
+
+ @Override
+ public void dropIndex() {
+ try {
+ Path[] indexPaths = new Path[]{getIndexByPartitionPath(metaClient), getIndexByFileIdPath(metaClient)};
+ for (Path indexPath : indexPaths) {
+ if (metaClient.getFs().exists(indexPath)) {
+ LOG.info("Dropping bootstrap index. Deleting file : " + indexPath);
+ metaClient.getFs().delete(indexPath);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ protected boolean checkIndex() {
+ return hasIndex;
+ }
+
+ /**
+ * HFile Based Index Reader.
+ */
+ public static class HFileBootstrapIndexReader extends BootstrapIndex.IndexReader {
+
+ // Base Path of external files.
+ private final String sourceBasePath;
+ // Well Known Paths for indices
+ private final String indexByPartitionPath;
+ private final String indexByFileIdPath;
+
+ // Index Readers
+ private transient HFile.Reader indexByPartitionReader;
+ private transient HFile.Reader indexByFileIdReader;
+
+ // Bootstrap Index Info
+ private transient BootstrapIndexInfo bootstrapIndexInfo;
+
+ public HFileBootstrapIndexReader(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ this.indexByPartitionPath = indexByPartitionPath.toString();
+ this.indexByFileIdPath = indexByFilePath.toString();
+ this.sourceBasePath = getIndexInfo().getSourceBasePath();
+ LOG.info("Loaded HFileBasedBootstrapIndex with source base path :" + sourceBasePath);
+ }
+
+ @Override
+ public BootstrapIndexInfo getIndexInfo() {
+ if (null == bootstrapIndexInfo) {
+ synchronized (this) {
+ if (null == bootstrapIndexInfo) {
+ try {
+ bootstrapIndexInfo = fetchBootstrapIndexInfo();
+ } catch (IOException ioe) {
+ throw new HoodieException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+ }
+ return bootstrapIndexInfo;
+ }
+
+ private BootstrapIndexInfo fetchBootstrapIndexInfo() throws IOException {
+ return TimelineMetadataUtils.deserializeAvroMetadata(
+ getIndexByPartitionReader().loadFileInfo().get(INDEX_INFO_KEY),
+ BootstrapIndexInfo.class);
+ }
+
+ private HFile.Reader getIndexByPartitionReader() {
+ if (null == indexByPartitionReader) {
+ synchronized (this) {
+ if (null == indexByPartitionReader) {
+ LOG.info("Opening partition index :" + indexByPartitionPath);
+ this.indexByPartitionReader =
+ createReader(indexByPartitionPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByPartitionReader;
+ }
+
+ private HFile.Reader getIndexByFileIdReader() {
+ if (null == indexByFileIdReader) {
+ synchronized (this) {
+ if (null == indexByFileIdReader) {
+ LOG.info("Opening fileId index :" + indexByFileIdPath);
+ this.indexByFileIdReader =
+ createReader(indexByFileIdPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByFileIdReader;
+ }
+
+ @Override
+ public List<String> getIndexedPartitions() {
+ HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+ return getAllKeys(scanner);
+ }
+
+ @Override
+ public List<String> getIndexedFileIds() {
+ HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+ return getAllKeys(scanner);
+ }
+
+ private List<String> getAllKeys(HFileScanner scanner) {
+ List<String> keys = new ArrayList<>();
+ try {
+ boolean available = scanner.seekTo();
+ while (available) {
+ keys.add(CellUtil.getCellKeyAsString(scanner.getKeyValue()));
+ available = scanner.next();
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+
+ return keys;
+ }
+
+ @Override
+ public List<BootstrapSourceFileMapping> getSourceFileMappingForPartition(String partition) {
+ try {
+ HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+ KeyValue keyValue = new KeyValue(Bytes.toBytes(getPartitionKey(partition)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+ if (scanner.seekTo(keyValue) == 0) {
+ ByteBuffer readValue = scanner.getValue();
+ byte[] valBytes = Bytes.toBytes(readValue);
+ BootstrapPartitionMetadata metadata =
+ TimelineMetadataUtils.deserializeAvroMetadata(valBytes, BootstrapPartitionMetadata.class);
+ return metadata.getHudiFileIdToSourceFile().entrySet().stream()
+ .map(e -> new BootstrapSourceFileMapping(sourceBasePath, metadata.getSourcePartitionPath(),
+ partition, e.getValue(), e.getKey())).collect(Collectors.toList());
+ } else {
+ LOG.info("No value found for partition key (" + partition + ")");
Review comment:
should this be in `info` level?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
+ } else {
+ hasIndex = false;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ private static String getPartitionKey(String partition) {
+ return "part=" + partition;
+ }
+
+ private static String getFileGroupKey(HoodieFileGroupId fileGroupId) {
+ return "part=" + fileGroupId.getPartitionPath() + ";fileid=" + fileGroupId.getFileId();
+ }
+
+ private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByPartitionFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ private static Path getIndexByFileIdPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByFileIdFolderNameFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ /**
+ * Helper method to create HFile Reader.
+ *
+ * @param hFilePath File Path
+ * @param conf Configuration
+ * @param fileSystem File System
+ */
+ private static HFile.Reader createReader(String hFilePath, Configuration conf, FileSystem fileSystem) {
+ try {
+ LOG.info("Opening HFile for reading :" + hFilePath);
+ HFile.Reader reader = HFile.createReader(fileSystem, new HFilePathForReader(hFilePath),
+ new CacheConfig(conf), conf);
+ return reader;
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public BootstrapIndex.IndexReader createReader() {
+ return new HFileBootstrapIndexReader(metaClient);
+ }
+
+ @Override
+ public BootstrapIndex.IndexWriter createWriter(String sourceBasePath) {
+ return new HFileBootstrapIndexWriter(sourceBasePath, metaClient);
+ }
+
+ @Override
+ public void dropIndex() {
+ try {
+ Path[] indexPaths = new Path[]{getIndexByPartitionPath(metaClient), getIndexByFileIdPath(metaClient)};
+ for (Path indexPath : indexPaths) {
+ if (metaClient.getFs().exists(indexPath)) {
+ LOG.info("Dropping bootstrap index. Deleting file : " + indexPath);
+ metaClient.getFs().delete(indexPath);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ protected boolean checkIndex() {
+ return hasIndex;
+ }
+
+ /**
+ * HFile Based Index Reader.
+ */
+ public static class HFileBootstrapIndexReader extends BootstrapIndex.IndexReader {
+
+ // Base Path of external files.
+ private final String sourceBasePath;
+ // Well Known Paths for indices
+ private final String indexByPartitionPath;
+ private final String indexByFileIdPath;
+
+ // Index Readers
+ private transient HFile.Reader indexByPartitionReader;
+ private transient HFile.Reader indexByFileIdReader;
+
+ // Bootstrap Index Info
+ private transient BootstrapIndexInfo bootstrapIndexInfo;
+
+ public HFileBootstrapIndexReader(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ this.indexByPartitionPath = indexByPartitionPath.toString();
+ this.indexByFileIdPath = indexByFilePath.toString();
+ this.sourceBasePath = getIndexInfo().getSourceBasePath();
+ LOG.info("Loaded HFileBasedBootstrapIndex with source base path :" + sourceBasePath);
+ }
+
+ @Override
+ public BootstrapIndexInfo getIndexInfo() {
+ if (null == bootstrapIndexInfo) {
+ synchronized (this) {
+ if (null == bootstrapIndexInfo) {
+ try {
+ bootstrapIndexInfo = fetchBootstrapIndexInfo();
+ } catch (IOException ioe) {
+ throw new HoodieException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+ }
+ return bootstrapIndexInfo;
+ }
+
+ private BootstrapIndexInfo fetchBootstrapIndexInfo() throws IOException {
+ return TimelineMetadataUtils.deserializeAvroMetadata(
+ getIndexByPartitionReader().loadFileInfo().get(INDEX_INFO_KEY),
+ BootstrapIndexInfo.class);
+ }
+
+ private HFile.Reader getIndexByPartitionReader() {
+ if (null == indexByPartitionReader) {
+ synchronized (this) {
+ if (null == indexByPartitionReader) {
+ LOG.info("Opening partition index :" + indexByPartitionPath);
+ this.indexByPartitionReader =
+ createReader(indexByPartitionPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByPartitionReader;
+ }
+
+ private HFile.Reader getIndexByFileIdReader() {
+ if (null == indexByFileIdReader) {
+ synchronized (this) {
+ if (null == indexByFileIdReader) {
+ LOG.info("Opening fileId index :" + indexByFileIdPath);
+ this.indexByFileIdReader =
+ createReader(indexByFileIdPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByFileIdReader;
+ }
+
+ @Override
+ public List<String> getIndexedPartitions() {
+ HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+ return getAllKeys(scanner);
+ }
+
+ @Override
+ public List<String> getIndexedFileIds() {
+ HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+ return getAllKeys(scanner);
+ }
+
+ private List<String> getAllKeys(HFileScanner scanner) {
+ List<String> keys = new ArrayList<>();
+ try {
+ boolean available = scanner.seekTo();
+ while (available) {
+ keys.add(CellUtil.getCellKeyAsString(scanner.getKeyValue()));
+ available = scanner.next();
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+
+ return keys;
+ }
+
+ @Override
+ public List<BootstrapSourceFileMapping> getSourceFileMappingForPartition(String partition) {
+ try {
+ HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+ KeyValue keyValue = new KeyValue(Bytes.toBytes(getPartitionKey(partition)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+ if (scanner.seekTo(keyValue) == 0) {
+ ByteBuffer readValue = scanner.getValue();
+ byte[] valBytes = Bytes.toBytes(readValue);
+ BootstrapPartitionMetadata metadata =
+ TimelineMetadataUtils.deserializeAvroMetadata(valBytes, BootstrapPartitionMetadata.class);
+ return metadata.getHudiFileIdToSourceFile().entrySet().stream()
+ .map(e -> new BootstrapSourceFileMapping(sourceBasePath, metadata.getSourcePartitionPath(),
+ partition, e.getValue(), e.getKey())).collect(Collectors.toList());
+ } else {
+ LOG.info("No value found for partition key (" + partition + ")");
+ return new ArrayList<>();
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public String getSourceBasePath() {
+ return sourceBasePath;
+ }
+
+ @Override
+ public Map<HoodieFileGroupId, BootstrapSourceFileMapping> getSourceFileMappingForFileIds(
+ List<HoodieFileGroupId> ids) {
+ Map<HoodieFileGroupId, BootstrapSourceFileMapping> result = new HashMap<>();
+ // Arrange input Keys in sorted order for 1 pass scan
+ List<HoodieFileGroupId> fileGroupIds = new ArrayList<>(ids);
+ Collections.sort(fileGroupIds);
+ try {
+ HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+ for (HoodieFileGroupId fileGroupId : fileGroupIds) {
+ KeyValue keyValue = new KeyValue(Bytes.toBytes(getFileGroupKey(fileGroupId)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+ if (scanner.seekTo(keyValue) == 0) {
+ ByteBuffer readValue = scanner.getValue();
+ byte[] valBytes = Bytes.toBytes(readValue);
+ BootstrapSourceFilePartitionInfo fileInfo = TimelineMetadataUtils.deserializeAvroMetadata(valBytes,
+ BootstrapSourceFilePartitionInfo.class);
+ BootstrapSourceFileMapping mapping = new BootstrapSourceFileMapping(sourceBasePath,
+ fileInfo.getSourcePartitionPath(), fileInfo.getHudiPartitionPath(), fileInfo.getSourceFileStatus(),
+ fileGroupId.getFileId());
+ result.put(fileGroupId, mapping);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ return result;
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (indexByPartitionReader != null) {
+ indexByPartitionReader.close(true);
+ indexByPartitionReader = null;
+ }
+ if (indexByFileIdReader != null) {
+ indexByFileIdReader.close(true);
+ indexByFileIdReader = null;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+
+ /**
+ * Boostrap Index Writer to build bootstrap index.
+ */
+ public static class HFileBootstrapIndexWriter extends BootstrapIndex.IndexWriter {
+
+ private final String sourceBasePath;
+ private final Path indexByPartitionPath;
+ private final Path indexByFileIdPath;
+ private HFile.Writer indexByPartitionWriter;
+ private HFile.Writer indexByFileIdWriter;
+
+ private boolean closed = false;
+ private int numPartitionKeysAdded = 0;
+ private int numFileIdKeysAdded = 0;
+
+ private final Map<String, List<BootstrapSourceFileMapping>> sourceFileMappings = new HashMap<>();
+
+ private HFileBootstrapIndexWriter(String sourceBasePath, HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ try {
+ metaClient.initializeBootstrapDirsIfNotExists();
+ this.sourceBasePath = sourceBasePath;
+ this.indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ this.indexByFileIdPath = getIndexByFileIdPath(metaClient);
+
+ if (metaClient.getFs().exists(indexByPartitionPath) || metaClient.getFs().exists(indexByFileIdPath)) {
+ String errMsg = "Previous version of bootstrap index exists. Partition Index Path :" + indexByPartitionPath
+ + ", FileId index Path :" + indexByFileIdPath;
+ LOG.info(errMsg);
+ throw new HoodieException(errMsg);
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ /**
+ * Append bootstrap index entries for next partitions in sorted order.
+ * @param hudiPartitionPath Hudi Partition Path
+ * @param sourcePartitionPath Source Partition Path
+ * @param bootstrapSourceFileMappings Bootstrap Source File to Hudi File Id mapping
+ */
+ private void writeNextPartition(String hudiPartitionPath, String sourcePartitionPath,
+ List<BootstrapSourceFileMapping> bootstrapSourceFileMappings) {
+ try {
+ LOG.info("Adding bootstrap partition Index entry for partition :" + hudiPartitionPath
+ + ", Source Partition :" + sourcePartitionPath + ", Num Entries :" + bootstrapSourceFileMappings.size());
+ LOG.info("ADDING entries :" + bootstrapSourceFileMappings);
+ BootstrapPartitionMetadata bootstrapPartitionMetadata = new BootstrapPartitionMetadata();
+ bootstrapPartitionMetadata.setSourcePartitionPath(sourcePartitionPath);
+ bootstrapPartitionMetadata.setHudiPartitionPath(hudiPartitionPath);
+ bootstrapPartitionMetadata.setHudiFileIdToSourceFile(
Review comment:
lets please not leak `hudi` into naming anywhere.. it will get very ugly with a mix of hudi/hoodie
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/BootstrapSourceFileMapping.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+
+/**
+ * Payload returned by Bootstrap Index.
+ */
+public class BootstrapSourceFileMapping implements Serializable, Comparable<BootstrapSourceFileMapping> {
Review comment:
rename: BootstrapFileMapping
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/HoodieBaseFile.java
##########
@@ -19,86 +19,63 @@
package org.apache.hudi.common.model;
import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.util.Option;
import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.Path;
-
-import java.io.Serializable;
-import java.util.Objects;
/**
- * Hoodie base file.
+ * Hoodie base file - Represents metadata about Hudi file in DFS.
+ * Supports APIs to get Hudi FileId, Commit Time and external file (for bootstrap).
*/
-public class HoodieBaseFile implements Serializable {
-
- private static final long serialVersionUID = 1L;
- private transient FileStatus fileStatus;
- private final String fullPath;
- private long fileLen;
+public class HoodieBaseFile extends BaseFile {
- public HoodieBaseFile(FileStatus fileStatus) {
- this.fileStatus = fileStatus;
- this.fullPath = fileStatus.getPath().toString();
- this.fileLen = fileStatus.getLen();
- }
+ /**
+ * In case of index-only bootstrap, this points to the actual data file.
Review comment:
lets remove references to specific bootstrap modes from here.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/BaseFile.java
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.Serializable;
+import java.util.Objects;
+
+/**
+ * Represents common metadata about base-file.
+ * A base file can be Hudi base file or even an external (non-hudi) base file too.
+ */
+public class BaseFile implements Serializable {
Review comment:
why is `HoodieBaseFile` not sufficient?
```
public class HoodieBaseFile implements Serializable {
private static final long serialVersionUID = 1L;
private transient FileStatus fileStatus;
private final String fullPath;
private long fileLen;
```
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
+
+ // Used as naming extensions.
+ public static final String BOOTSTRAP_INDEX_FILE_TYPE = "hfile";
+
+ // Additional Metadata written to HFiles.
+ public static final byte[] INDEX_INFO_KEY = Bytes.toBytes("INDEX_INFO");
+
+ // Flag to idenitfy if Bootstrap Index is empty or not
+ private final boolean hasIndex;
+
+ public HFileBasedBootstrapIndex(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ try {
+ if (metaClient.getFs().exists(indexByPartitionPath) && metaClient.getFs().exists(indexByFilePath)) {
+ hasIndex = true;
+ } else {
+ hasIndex = false;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ private static String getPartitionKey(String partition) {
+ return "part=" + partition;
+ }
+
+ private static String getFileGroupKey(HoodieFileGroupId fileGroupId) {
+ return "part=" + fileGroupId.getPartitionPath() + ";fileid=" + fileGroupId.getFileId();
+ }
+
+ private static Path getIndexByPartitionPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByPartitionFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ private static Path getIndexByFileIdPath(HoodieTableMetaClient metaClient) {
+ return new Path(metaClient.getBootstrapIndexByFileIdFolderNameFolderPath(),
+ FSUtils.makeBootstrapIndexFileName(HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, BOOTSTRAP_INDEX_FILE_ID,
+ BOOTSTRAP_INDEX_FILE_TYPE));
+ }
+
+ /**
+ * Helper method to create HFile Reader.
+ *
+ * @param hFilePath File Path
+ * @param conf Configuration
+ * @param fileSystem File System
+ */
+ private static HFile.Reader createReader(String hFilePath, Configuration conf, FileSystem fileSystem) {
+ try {
+ LOG.info("Opening HFile for reading :" + hFilePath);
+ HFile.Reader reader = HFile.createReader(fileSystem, new HFilePathForReader(hFilePath),
+ new CacheConfig(conf), conf);
+ return reader;
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public BootstrapIndex.IndexReader createReader() {
+ return new HFileBootstrapIndexReader(metaClient);
+ }
+
+ @Override
+ public BootstrapIndex.IndexWriter createWriter(String sourceBasePath) {
+ return new HFileBootstrapIndexWriter(sourceBasePath, metaClient);
+ }
+
+ @Override
+ public void dropIndex() {
+ try {
+ Path[] indexPaths = new Path[]{getIndexByPartitionPath(metaClient), getIndexByFileIdPath(metaClient)};
+ for (Path indexPath : indexPaths) {
+ if (metaClient.getFs().exists(indexPath)) {
+ LOG.info("Dropping bootstrap index. Deleting file : " + indexPath);
+ metaClient.getFs().delete(indexPath);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ protected boolean checkIndex() {
+ return hasIndex;
+ }
+
+ /**
+ * HFile Based Index Reader.
+ */
+ public static class HFileBootstrapIndexReader extends BootstrapIndex.IndexReader {
+
+ // Base Path of external files.
+ private final String sourceBasePath;
+ // Well Known Paths for indices
+ private final String indexByPartitionPath;
+ private final String indexByFileIdPath;
+
+ // Index Readers
+ private transient HFile.Reader indexByPartitionReader;
+ private transient HFile.Reader indexByFileIdReader;
+
+ // Bootstrap Index Info
+ private transient BootstrapIndexInfo bootstrapIndexInfo;
+
+ public HFileBootstrapIndexReader(HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ Path indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ Path indexByFilePath = getIndexByFileIdPath(metaClient);
+ this.indexByPartitionPath = indexByPartitionPath.toString();
+ this.indexByFileIdPath = indexByFilePath.toString();
+ this.sourceBasePath = getIndexInfo().getSourceBasePath();
+ LOG.info("Loaded HFileBasedBootstrapIndex with source base path :" + sourceBasePath);
+ }
+
+ @Override
+ public BootstrapIndexInfo getIndexInfo() {
+ if (null == bootstrapIndexInfo) {
+ synchronized (this) {
+ if (null == bootstrapIndexInfo) {
+ try {
+ bootstrapIndexInfo = fetchBootstrapIndexInfo();
+ } catch (IOException ioe) {
+ throw new HoodieException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+ }
+ return bootstrapIndexInfo;
+ }
+
+ private BootstrapIndexInfo fetchBootstrapIndexInfo() throws IOException {
+ return TimelineMetadataUtils.deserializeAvroMetadata(
+ getIndexByPartitionReader().loadFileInfo().get(INDEX_INFO_KEY),
+ BootstrapIndexInfo.class);
+ }
+
+ private HFile.Reader getIndexByPartitionReader() {
+ if (null == indexByPartitionReader) {
+ synchronized (this) {
+ if (null == indexByPartitionReader) {
+ LOG.info("Opening partition index :" + indexByPartitionPath);
+ this.indexByPartitionReader =
+ createReader(indexByPartitionPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByPartitionReader;
+ }
+
+ private HFile.Reader getIndexByFileIdReader() {
+ if (null == indexByFileIdReader) {
+ synchronized (this) {
+ if (null == indexByFileIdReader) {
+ LOG.info("Opening fileId index :" + indexByFileIdPath);
+ this.indexByFileIdReader =
+ createReader(indexByFileIdPath, metaClient.getHadoopConf(), metaClient.getFs());
+ }
+ }
+ }
+ return indexByFileIdReader;
+ }
+
+ @Override
+ public List<String> getIndexedPartitions() {
+ HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+ return getAllKeys(scanner);
+ }
+
+ @Override
+ public List<String> getIndexedFileIds() {
+ HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+ return getAllKeys(scanner);
+ }
+
+ private List<String> getAllKeys(HFileScanner scanner) {
+ List<String> keys = new ArrayList<>();
+ try {
+ boolean available = scanner.seekTo();
+ while (available) {
+ keys.add(CellUtil.getCellKeyAsString(scanner.getKeyValue()));
+ available = scanner.next();
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+
+ return keys;
+ }
+
+ @Override
+ public List<BootstrapSourceFileMapping> getSourceFileMappingForPartition(String partition) {
+ try {
+ HFileScanner scanner = getIndexByPartitionReader().getScanner(true, true);
+ KeyValue keyValue = new KeyValue(Bytes.toBytes(getPartitionKey(partition)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+ if (scanner.seekTo(keyValue) == 0) {
+ ByteBuffer readValue = scanner.getValue();
+ byte[] valBytes = Bytes.toBytes(readValue);
+ BootstrapPartitionMetadata metadata =
+ TimelineMetadataUtils.deserializeAvroMetadata(valBytes, BootstrapPartitionMetadata.class);
+ return metadata.getHudiFileIdToSourceFile().entrySet().stream()
+ .map(e -> new BootstrapSourceFileMapping(sourceBasePath, metadata.getSourcePartitionPath(),
+ partition, e.getValue(), e.getKey())).collect(Collectors.toList());
+ } else {
+ LOG.info("No value found for partition key (" + partition + ")");
+ return new ArrayList<>();
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ @Override
+ public String getSourceBasePath() {
+ return sourceBasePath;
+ }
+
+ @Override
+ public Map<HoodieFileGroupId, BootstrapSourceFileMapping> getSourceFileMappingForFileIds(
+ List<HoodieFileGroupId> ids) {
+ Map<HoodieFileGroupId, BootstrapSourceFileMapping> result = new HashMap<>();
+ // Arrange input Keys in sorted order for 1 pass scan
+ List<HoodieFileGroupId> fileGroupIds = new ArrayList<>(ids);
+ Collections.sort(fileGroupIds);
+ try {
+ HFileScanner scanner = getIndexByFileIdReader().getScanner(true, true);
+ for (HoodieFileGroupId fileGroupId : fileGroupIds) {
+ KeyValue keyValue = new KeyValue(Bytes.toBytes(getFileGroupKey(fileGroupId)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, new byte[0]);
+ if (scanner.seekTo(keyValue) == 0) {
+ ByteBuffer readValue = scanner.getValue();
+ byte[] valBytes = Bytes.toBytes(readValue);
+ BootstrapSourceFilePartitionInfo fileInfo = TimelineMetadataUtils.deserializeAvroMetadata(valBytes,
+ BootstrapSourceFilePartitionInfo.class);
+ BootstrapSourceFileMapping mapping = new BootstrapSourceFileMapping(sourceBasePath,
+ fileInfo.getSourcePartitionPath(), fileInfo.getHudiPartitionPath(), fileInfo.getSourceFileStatus(),
+ fileGroupId.getFileId());
+ result.put(fileGroupId, mapping);
+ }
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ return result;
+ }
+
+ @Override
+ public void close() {
+ try {
+ if (indexByPartitionReader != null) {
+ indexByPartitionReader.close(true);
+ indexByPartitionReader = null;
+ }
+ if (indexByFileIdReader != null) {
+ indexByFileIdReader.close(true);
+ indexByFileIdReader = null;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+
+ /**
+ * Boostrap Index Writer to build bootstrap index.
+ */
+ public static class HFileBootstrapIndexWriter extends BootstrapIndex.IndexWriter {
+
+ private final String sourceBasePath;
+ private final Path indexByPartitionPath;
+ private final Path indexByFileIdPath;
+ private HFile.Writer indexByPartitionWriter;
+ private HFile.Writer indexByFileIdWriter;
+
+ private boolean closed = false;
+ private int numPartitionKeysAdded = 0;
+ private int numFileIdKeysAdded = 0;
+
+ private final Map<String, List<BootstrapSourceFileMapping>> sourceFileMappings = new HashMap<>();
+
+ private HFileBootstrapIndexWriter(String sourceBasePath, HoodieTableMetaClient metaClient) {
+ super(metaClient);
+ try {
+ metaClient.initializeBootstrapDirsIfNotExists();
+ this.sourceBasePath = sourceBasePath;
+ this.indexByPartitionPath = getIndexByPartitionPath(metaClient);
+ this.indexByFileIdPath = getIndexByFileIdPath(metaClient);
+
+ if (metaClient.getFs().exists(indexByPartitionPath) || metaClient.getFs().exists(indexByFileIdPath)) {
+ String errMsg = "Previous version of bootstrap index exists. Partition Index Path :" + indexByPartitionPath
+ + ", FileId index Path :" + indexByFileIdPath;
+ LOG.info(errMsg);
+ throw new HoodieException(errMsg);
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ /**
+ * Append bootstrap index entries for next partitions in sorted order.
+ * @param hudiPartitionPath Hudi Partition Path
+ * @param sourcePartitionPath Source Partition Path
+ * @param bootstrapSourceFileMappings Bootstrap Source File to Hudi File Id mapping
+ */
+ private void writeNextPartition(String hudiPartitionPath, String sourcePartitionPath,
+ List<BootstrapSourceFileMapping> bootstrapSourceFileMappings) {
+ try {
+ LOG.info("Adding bootstrap partition Index entry for partition :" + hudiPartitionPath
+ + ", Source Partition :" + sourcePartitionPath + ", Num Entries :" + bootstrapSourceFileMappings.size());
+ LOG.info("ADDING entries :" + bootstrapSourceFileMappings);
+ BootstrapPartitionMetadata bootstrapPartitionMetadata = new BootstrapPartitionMetadata();
+ bootstrapPartitionMetadata.setSourcePartitionPath(sourcePartitionPath);
+ bootstrapPartitionMetadata.setHudiPartitionPath(hudiPartitionPath);
+ bootstrapPartitionMetadata.setHudiFileIdToSourceFile(
+ bootstrapSourceFileMappings.stream().map(m -> Pair.of(m.getHudiFileId(),
+ m.getSourceFileStatus())).collect(Collectors.toMap(Pair::getKey, Pair::getValue)));
+ Option<byte[]> bytes = TimelineMetadataUtils.serializeAvroMetadata(bootstrapPartitionMetadata,
+ BootstrapPartitionMetadata.class);
+ if (bytes.isPresent()) {
+ indexByPartitionWriter
+ .append(new KeyValue(Bytes.toBytes(getPartitionKey(hudiPartitionPath)), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put, bytes.get()));
+ numPartitionKeysAdded++;
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Write next source file to hudi file-id. Entries are expected to be appended in hudi file-group id
+ * order.
+ * @param mapping boostrap source file mapping.
+ */
+ private void writeNextSourceFileMapping(BootstrapSourceFileMapping mapping) {
+ try {
+ BootstrapSourceFilePartitionInfo srcFilePartitionInfo = new BootstrapSourceFilePartitionInfo();
+ srcFilePartitionInfo.setHudiPartitionPath(mapping.getHudiPartitionPath());
+ srcFilePartitionInfo.setSourcePartitionPath(mapping.getSourcePartitionPath());
+ srcFilePartitionInfo.setSourceFileStatus(mapping.getSourceFileStatus());
+ KeyValue kv = new KeyValue(getFileGroupKey(mapping.getFileGroupId()).getBytes(), new byte[0], new byte[0],
+ HConstants.LATEST_TIMESTAMP, KeyValue.Type.Put,
+ TimelineMetadataUtils.serializeAvroMetadata(srcFilePartitionInfo,
+ BootstrapSourceFilePartitionInfo.class).get());
+ indexByFileIdWriter.append(kv);
+ numFileIdKeysAdded++;
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Commit bootstrap index entries. Appends Metadata and closes write handles.
+ */
+ private void commit() {
Review comment:
just want to ensure that overall we will be able to recover from partial crashes during this phase, where we write one index and die before writing the other and so on?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/BootstrapSourceFileMapping.java
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import java.io.Serializable;
+import java.util.Objects;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+
+/**
+ * Payload returned by Bootstrap Index.
+ */
+public class BootstrapSourceFileMapping implements Serializable, Comparable<BootstrapSourceFileMapping> {
+
+ private final String sourceBasePath;
Review comment:
consistent use of either external, source or bootstrap. let's pick one. Please apply this throughout this PR..
My choice of language for this PR : bootstrap
- bootstrap : when talking about the process
- bootstrappedFile/bootstrappedPath : when talking about a file that was bootstrapped
- bootstrapPartitionPath
& so on..
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/CompactionOperation.java
##########
@@ -111,9 +114,17 @@ public HoodieFileGroupId getFileGroupId() {
return id;
}
+ public Option<String> getExternalFilePath() {
Review comment:
you probably can guess what I am going to say here :)
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/CompactionOperation.java
##########
@@ -44,18 +44,20 @@
private Option<String> dataFileName;
private HoodieFileGroupId id;
private Map<String, Double> metrics;
+ private Option<String> externalFilePath;
// Only for serialization/de-serialization
@Deprecated
public CompactionOperation() {}
public CompactionOperation(String fileId, String partitionPath, String baseInstantTime,
Option<String> dataFileCommitTime, List<String> deltaFileNames, Option<String> dataFileName,
- Map<String, Double> metrics) {
+ Option<String> externalFilePath, Map<String, Double> metrics) {
Review comment:
why bake this into the plan as opposed to looking up bootstrap index during compaction? does nt leak bootstrap into the compaction planning etc?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java
##########
@@ -81,6 +81,13 @@
String INVALID_INSTANT_TS = "0";
+ // Instant corresponding to pristine state of the table after its creation
Review comment:
we should add a check elsewhere that throws an error if the user attempts to supply these as commit times (by passing in a value, like say what our tests do)
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -288,6 +329,12 @@ public Operation convert(String value) throws ParameterException {
+ "Use this field only when switching source, for example, from DFS source to Kafka Source.")
public String initialCheckpointProvider = null;
+ @Parameter(names = {"--run-bootstrap"}, description = "Run bootstrap if bootstrap index is not found")
Review comment:
please make sure the description is user understandable..
I dont know if we should have a separate option for --run-bootstrap.. (see comment above)
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -73,6 +73,12 @@
public static final String METAFOLDER_NAME = ".hoodie";
public static final String TEMPFOLDER_NAME = METAFOLDER_NAME + File.separator + ".temp";
public static final String AUXILIARYFOLDER_NAME = METAFOLDER_NAME + File.separator + ".aux";
+ public static final String BOOTSTRAP_INDEX_ROOT_FOLDER_PATH = AUXILIARYFOLDER_NAME + File.separator + ".bootstrap";
Review comment:
should we keep this inside `.aux` or directly under `.hoodie`.. what are the tradeoffs herE?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -122,20 +145,37 @@ public void shutdownGracefully() {
* @throws Exception
*/
public void sync() throws Exception {
- if (cfg.continuousMode) {
- deltaSyncService.start(this::onDeltaSyncShutdown);
- deltaSyncService.waitForShutdown();
- LOG.info("Delta Sync shutting down");
+ if (bootstrapExecutor.isPresent()) {
Review comment:
can't we just have it so that the first run of DeltaSync will perform bootstrap if configured and move on?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -207,6 +213,20 @@ public String getMetaAuxiliaryPath() {
return basePath + File.separator + AUXILIARYFOLDER_NAME;
}
+ /**
+ * @return Bootstrap Index By Partition Folder
+ */
+ public String getBootstrapIndexByPartitionFolderPath() {
Review comment:
general thought (may be should have written this above).. why have two separate files? why not just write two column families or key spaces into the same HFile.. (amortizes on need to open two files on query side for everything .. ) if you say prefix the key with partitionPath then you will get lucky with caching probably..
##########
File path: scripts/run_travis_tests.sh
##########
@@ -17,13 +17,14 @@
# limitations under the License.
mode=$1
+modules=$2
sparkVersion=2.4.4
hadoopVersion=2.7
if [ "$mode" = "unit" ];
then
echo "Running Unit Tests"
- mvn test -DskipITs=true -B
+ mvn test -DskipITs=true -pl "$modules" -B
Review comment:
let's break these into a separate PR and merge ahead? if we see flakiness, then we wont know the clear cause if we bundle this together.
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieCommonTestHarness.java
##########
@@ -42,7 +43,13 @@
* Initializes basePath.
*/
protected void initPath() {
- this.basePath = tempDir.toAbsolutePath().toString();
+ try {
+ java.nio.file.Path basePath = tempDir.resolve("dataset");
Review comment:
why this change?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java
##########
@@ -190,7 +190,7 @@ public static MessageType readSchema(Configuration configuration, Path parquetFi
}
public static Schema readAvroSchema(Configuration configuration, Path parquetFilePath) {
- return new AvroSchemaConverter().convert(readSchema(configuration, parquetFilePath));
+ return new AvroSchemaConverter(configuration).convert(readSchema(configuration, parquetFilePath));
Review comment:
merge as a separate fix?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/NullSkeletonRecordReader.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import java.io.IOException;
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hudi.common.model.HoodieRecord;
+
+public class NullSkeletonRecordReader implements RecordReader<NullWritable, ArrayWritable> {
Review comment:
why do we need this
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/FileSystemViewStorageConfig.java
##########
@@ -41,6 +41,8 @@
public static final String FILESYSTEM_VIEW_SPILLABLE_MEM = "hoodie.filesystem.view.spillable.mem";
public static final String FILESYSTEM_VIEW_PENDING_COMPACTION_MEM_FRACTION =
"hoodie.filesystem.view.spillable.compaction.mem.fraction";
+ public static final String FILESYSTEM_VIEW_EXTERNAL_DATA_FILE_FRACTION =
Review comment:
doc on what this is
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/checkpointing/InitialCheckpointFromAnotherHoodieTimelineProvider.java
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.checkpointing;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.exception.HoodieException;
+
+import java.io.IOException;
+import java.util.Objects;
+
+import static org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.CHECKPOINT_KEY;
+
+/**
+ * This is used to set a checkpoint from latest commit of another (mirror) hudi dataset.
+ * Used by integration test.
+ */
+public class InitialCheckpointFromAnotherHoodieTimelineProvider extends InitialCheckPointProvider {
Review comment:
then should it live somewhere inside src/test?
##########
File path: packaging/hudi-hadoop-mr-bundle/pom.xml
##########
@@ -145,5 +169,37 @@
<artifactId>avro</artifactId>
<scope>compile</scope>
</dependency>
+
+ <dependency>
+ <groupId>org.apache.htrace</groupId>
+ <artifactId>htrace-core</artifactId>
+ <version>${htrace.version}</version>
+ <scope>compile</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-server</artifactId>
+ <version>${hbase.version}</version>
+ <scope>compile</scope>
+ <exclusions>
Review comment:
do we need the exclusions per se?we are specifically including packages above when bundling right
##########
File path: hudi-spark/src/test/java/org/apache/hudi/client/TestBootstrap.java
##########
@@ -0,0 +1,526 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client;
+
+import org.apache.hudi.DataSourceWriteOptions;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.selector.FullBootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieTableType;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieBootstrapConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.hadoop.HoodieParquetInputFormat;
+import org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.io.DoubleWritable;
+import org.apache.hudi.index.HoodieIndex.IndexType;
+import org.apache.hudi.keygen.NonpartitionedKeyGenerator;
+import org.apache.hudi.keygen.SimpleKeyGenerator;
+import org.apache.hudi.testutils.HoodieClientTestBase;
+import org.apache.hudi.testutils.HoodieMergeOnReadTestUtils;
+import org.apache.hudi.testutils.HoodieTestDataGenerator;
+import org.apache.hudi.testutils.TestRawTripPayload;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+
+import org.apache.spark.sql.SaveMode;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.IOException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Set;
+import java.util.Map;
+import java.util.Random;
+import java.util.Spliterators;
+import java.util.stream.Collectors;
+import java.util.stream.StreamSupport;
+
+import static java.util.stream.Collectors.mapping;
+import static java.util.stream.Collectors.toList;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Tests Bootstrap Client functionality.
+ */
+public class TestBootstrap extends HoodieClientTestBase {
+
+ @TempDir
+ public java.nio.file.Path tmpFolder;
+
+ protected String srcPath = null;
+
+ private HoodieParquetInputFormat roInputFormat;
+ private JobConf roJobConf;
+
+ private HoodieParquetRealtimeInputFormat rtInputFormat;
+ private JobConf rtJobConf;
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initResources();
+
+ srcPath = tmpFolder.toAbsolutePath().toString() + "/data";
+
+ // initialize parquet input format
+ reloadInputFormats();
+ }
+
+ private void reloadInputFormats() {
+ // initialize parquet input format
+ roInputFormat = new HoodieParquetInputFormat();
+ roJobConf = new JobConf(jsc.hadoopConfiguration());
+ roInputFormat.setConf(roJobConf);
+
+ rtInputFormat = new HoodieParquetRealtimeInputFormat();
+ rtJobConf = new JobConf(jsc.hadoopConfiguration());
+ rtInputFormat.setConf(rtJobConf);
+ }
+
+ @AfterEach
+ public void tearDown() throws Exception {
+ cleanupClients();
+ cleanupTestDataGenerator();
+ cleanupFileSystem();
+ // Do NOT cleanup Spark Context as it is being provided and reused by other tests in hudi-spark.
+ // See HoodieClientTestHarness.initSparkContexts()
+ }
+
+ public Schema generateNewDataSetAndReturnSchema(double timestamp, int numRecords, List<String> partitionPaths,
+ String srcPath) throws Exception {
+ boolean isPartitioned = partitionPaths != null && !partitionPaths.isEmpty();
+ Dataset<Row> df = HoodieTestDataGenerator.generateTestRawTripDataset(timestamp,
+ numRecords, partitionPaths, jsc, sqlContext);
+ df.printSchema();
+ if (isPartitioned) {
+ df.write().partitionBy("datestr").format("parquet").mode(SaveMode.Overwrite).save(srcPath);
Review comment:
does this test use Spark dataset API for other than this? should this test live in `hudi-spark` or `hudi-client`?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -288,6 +329,12 @@ public Operation convert(String value) throws ParameterException {
+ "Use this field only when switching source, for example, from DFS source to Kafka Source.")
public String initialCheckpointProvider = null;
+ @Parameter(names = {"--run-bootstrap"}, description = "Run bootstrap if bootstrap index is not found")
+ public Boolean runBootstrap = false;
+
+ @Parameter(names = {"--bootstrap-index-class"}, description = "subclass of BootstrapIndex")
Review comment:
how do I specify the bootstrapBasePath?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieColumnProjectionUtils.java
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
+import org.apache.hudi.common.util.CollectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.io.IOConstants;
+import org.apache.hadoop.hive.serde2.ColumnProjectionUtils;
+import org.apache.hadoop.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Utility funcitons copied from Hive ColumnProjectionUtils.java.
+ * Needed to copy as we see NoSuchMethod errors when directly using these APIs with/without Spark.
Review comment:
what was the root cause for these errors?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieColumnStichingRecordReader.java
##########
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.RecordReader;
+
+import java.io.IOException;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Stitches 2 record reader returned rows and presents a concatenated view to clients.
+ */
+public class HoodieColumnStichingRecordReader implements RecordReader<NullWritable, ArrayWritable> {
Review comment:
somehow we need to make the naming be aware of bootstrap.. since thats what it does really.
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/FileStatusWithExternalBaseFile.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+/**
+ * Sub-Type of File Status tracking both skeleton and external base file's status.
Review comment:
this skeleton file is not used anywhere else except in some sporadic comments.. might want to consider either using it fully.. or explain it differently
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/ExternalBaseFileSplit.java
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.mapred.FileSplit;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+
+
+/**
+ * Sub-type of File Split which encapsulates both skeleton and external base file splits.
+ */
+public class ExternalBaseFileSplit extends FileSplit {
+
+ private FileSplit externalFileSplit;
+
+ public ExternalBaseFileSplit() {
+ super();
+ }
+
+ public ExternalBaseFileSplit(FileSplit baseSplit, FileSplit externalFileSplit)
+ throws IOException {
+ super(baseSplit.getPath(), baseSplit.getStart(), baseSplit.getLength(), baseSplit.getLocations());
+ this.externalFileSplit = externalFileSplit;
+ }
+
+ public FileSplit getExternalFileSplit() {
+ return externalFileSplit;
+ }
+
+ private static void writeString(String str, DataOutput out) throws IOException {
Review comment:
should sit some StringUtils or so?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodiePathWithExternalFileStatus.java
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Hacky Workaround !!!
Review comment:
I am not following this.. lets unhack it before we merge :)
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
##########
@@ -335,6 +357,53 @@ public Configuration getConf() {
// ParquetInputFormat.setFilterPredicate(job, predicate);
// clearOutExistingPredicate(job);
// }
+ if (split instanceof ExternalBaseFileSplit) {
Review comment:
this block is too large.. break it into methods?
##########
File path: hudi-spark/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -207,7 +207,8 @@ object DataSourceWriteOptions {
*/
val HIVE_STYLE_PARTITIONING_OPT_KEY = "hoodie.datasource.write.hive_style_partitioning"
val DEFAULT_HIVE_STYLE_PARTITIONING_OPT_VAL = "false"
-
+ val URL_ENCODE_PARTITIONING_OPT_KEY = "hoodie.datasource.write.partitionpath.urlencode"
+ val DEFAULT_URL_ENCODE_PARTITIONING_OPT_VAL = "false"
Review comment:
how come we dont have options to specify bootstrap base path etc?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieParquetInputFormat.java
##########
@@ -335,6 +357,53 @@ public Configuration getConf() {
// ParquetInputFormat.setFilterPredicate(job, predicate);
// clearOutExistingPredicate(job);
// }
+ if (split instanceof ExternalBaseFileSplit) {
+ ExternalBaseFileSplit eSplit = (ExternalBaseFileSplit)split;
+ String[] rawColNames = HoodieColumnProjectionUtils.getReadColumnNames(job);
+ List<Integer> rawColIds = HoodieColumnProjectionUtils.getReadColumnIDs(job);
+ List<Pair<Integer, String>> projectedColsWithIndex =
+ IntStream.range(0, rawColIds.size()).mapToObj(idx -> Pair.of(rawColIds.get(idx), rawColNames[idx]))
+ .collect(Collectors.toList());
+
+ List<Pair<Integer, String>> hoodieColsProjected = projectedColsWithIndex.stream()
+ .filter(idxWithName -> HoodieRecord.HOODIE_META_COLUMNS.contains(idxWithName.getValue()))
+ .collect(Collectors.toList());
+ List<Pair<Integer, String>> externalColsProjected = projectedColsWithIndex.stream()
+ .filter(idxWithName -> !HoodieRecord.HOODIE_META_COLUMNS.contains(idxWithName.getValue())
+ && !VirtualColumn.VIRTUAL_COLUMN_NAMES.contains(idxWithName.getValue()))
+ .collect(Collectors.toList());
+
+ // This always matches hive table description
+ List<Pair<String, String>> colNameWithTypes = HoodieColumnProjectionUtils.getIOColumnNameAndTypes(job);
+ List<Pair<String, String>> colNamesWithTypesForExternal = colNameWithTypes.stream()
+ .filter(p -> !HoodieRecord.HOODIE_META_COLUMNS.contains(p.getKey())).collect(Collectors.toList());
+ LOG.info("colNameWithTypes =" + colNameWithTypes + ", Num Entries =" + colNameWithTypes.size());
+ if (hoodieColsProjected.isEmpty()) {
+ return super.getRecordReader(eSplit.getExternalFileSplit(), job, reporter);
+ } else if (externalColsProjected.isEmpty()) {
+ return super.getRecordReader(split, job, reporter);
+ } else {
+ FileSplit rightSplit = eSplit.getExternalFileSplit();
+ // Hive PPD works at row-group level and only enabled when hive.optimize.index.filter=true;
Review comment:
is this code borrowed from somewhere else?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieParquetRealtimeInputFormat.java
##########
@@ -252,16 +272,24 @@ private static void cleanProjectionColumnIds(Configuration conf) {
LOG.info("Creating record reader with readCols :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR)
+ ", Ids :" + jobConf.get(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR));
- // sanity check
- ValidationUtils.checkArgument(split instanceof HoodieRealtimeFileSplit,
- "HoodieRealtimeRecordReader can only work on HoodieRealtimeFileSplit and not with " + split);
- return new HoodieRealtimeRecordReader((HoodieRealtimeFileSplit) split, jobConf,
- super.getRecordReader(split, jobConf, reporter));
+ // sanity check
+ ValidationUtils.checkArgument(split instanceof RealtimeSplit,
+ "HoodieRealtimeRecordReader can only work on RealtimeSplit and not with " + split);
+ JobConf jobConf1 = new JobConf(jobConf);
+ JobConf jobConf2 = new JobConf(jobConf);
+ if (split instanceof ExternalBaseFileSplit) {
Review comment:
is there a nicer way of doing this than having these instanceof checks everywher?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
##########
@@ -117,6 +124,16 @@ protected void refreshTimeline(HoodieTimeline visibleActiveTimeline) {
// Group by partition for efficient updates for both InMemory and DiskBased stuctures.
fileGroups.stream().collect(Collectors.groupingBy(HoodieFileGroup::getPartitionPath)).forEach((partition, value) -> {
if (!isPartitionAvailableInStore(partition)) {
+ if (bootstrapIndex.isIndexAvailable()) {
Review comment:
this will incur a fs.exists() everytime? I suggest we add a flag to `hoodie.properties` to denote whether a table was bootstrapped or not and just use that from the metaClient?
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HoodieColumnStichingRecordReader.java
##########
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.io.ArrayWritable;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.RecordReader;
+
+import java.io.IOException;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+/**
+ * Stitches 2 record reader returned rows and presents a concatenated view to clients.
+ */
+public class HoodieColumnStichingRecordReader implements RecordReader<NullWritable, ArrayWritable> {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieColumnStichingRecordReader.class);
+
+ private final RecordReader<NullWritable, ArrayWritable> leftColsRecordReader;
+ private final RecordReader<NullWritable, ArrayWritable> rightColsRecordReader;
+ private final int numLeftColumns;
+ private final int numRightColumns;
+ private final ArrayWritable values;
+ private final boolean validate;
+
+ public HoodieColumnStichingRecordReader(RecordReader<NullWritable, ArrayWritable> left,
+ int numLeftColumns, RecordReader<NullWritable, ArrayWritable> right, int numRightColumns, boolean validate) {
+ this.leftColsRecordReader = left;
+ this.rightColsRecordReader = right;
+ this.validate = validate;
+ this.numLeftColumns = numLeftColumns;
+ this.numRightColumns = numRightColumns;
+
+ ArrayWritable rightW = rightColsRecordReader.createValue();
+ int numColumns = numLeftColumns + numRightColumns;
+ if (rightW.getValueClass() != null) {
+ values = new ArrayWritable(rightW.getValueClass(), new Writable[numColumns]);
+ } else {
+ String[] vals = IntStream.range(0, numColumns).mapToObj(idx -> "").collect(Collectors.toList())
+ .toArray(new String[0]);
+ values = new ArrayWritable(vals);
+ }
+ LOG.info("Total ArrayWritable Length :" + values.get().length);
+ }
+
+ @Override
+ public boolean next(NullWritable key, ArrayWritable value) throws IOException {
+ ArrayWritable left = leftColsRecordReader.createValue();
+ ArrayWritable right = rightColsRecordReader.createValue();
+
+ boolean hasMoreOnLeft = leftColsRecordReader.next(leftColsRecordReader.createKey(), left);
+ boolean hasMoreOnRight = rightColsRecordReader.next(rightColsRecordReader.createKey(), right);
+ if (validate) {
+ ValidationUtils.checkArgument(hasMoreOnLeft == hasMoreOnRight);
+ }
+ //LOG.info("Left expected length :" + numLeftColumns + ", Got :" + left.get().length);
+ for (int i = 0; i < numLeftColumns; i++) {
+ value.get()[i] = left.get()[i];
+ }
+ //LOG.info("Right expected length :" + numRightColumns + ", Got :" + right.get().length);
Review comment:
remove?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #1678: [WIP] [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#issuecomment-636276146
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=h1) Report
> Merging [#1678](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/3574a89232ad8573d1f76b27f127cc2ab9195e38&el=desc) will **decrease** coverage by `1.06%`.
> The diff coverage is `6.93%`.
[](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1678 +/- ##
============================================
- Coverage 18.22% 17.16% -1.07%
- Complexity 857 888 +31
============================================
Files 348 388 +40
Lines 15332 16758 +1426
Branches 1523 1644 +121
============================================
+ Hits 2794 2876 +82
- Misses 12181 13514 +1333
- Partials 357 368 +11
```
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...java/org/apache/hudi/client/HoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `18.77% <0.00%> (-1.61%)` | `12.00 <0.00> (ø)` | |
| [.../hudi/client/bootstrap/BootstrapRecordPayload.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Cb290c3RyYXBSZWNvcmRQYXlsb2FkLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...lient/bootstrap/BootstrapSourceSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Cb290c3RyYXBTb3VyY2VTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...he/hudi/client/bootstrap/BootstrapWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Cb290c3RyYXBXcml0ZVN0YXR1cy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...i/client/bootstrap/FullBootstrapInputProvider.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9GdWxsQm9vdHN0cmFwSW5wdXRQcm92aWRlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...e/hudi/client/bootstrap/HoodieBootstrapClient.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Ib29kaWVCb290c3RyYXBDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...lient/bootstrap/MetadataBootstrapKeyGenerator.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9NZXRhZGF0YUJvb3RzdHJhcEtleUdlbmVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...ient/bootstrap/selector/BootstrapModeSelector.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9zZWxlY3Rvci9Cb290c3RyYXBNb2RlU2VsZWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...bootstrap/selector/BootstrapRegexModeSelector.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9zZWxlY3Rvci9Cb290c3RyYXBSZWdleE1vZGVTZWxlY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [.../bootstrap/selector/FullBootstrapModeSelector.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9zZWxlY3Rvci9GdWxsQm9vdHN0cmFwTW9kZVNlbGVjdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| ... and [128 more](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=footer). Last update [3574a89...8824a93](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r461755543
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/MergeHelper.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.commit;
+
+import java.io.ByteArrayOutputStream;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.utils.MergingParquetIterator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieMergeHandle;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.schema.MessageType;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Helper to read records from previous version of parquet and run Merge.
+ */
+public class MergeHelper {
+
+ /**
+ * Read records from previous version of base file and merge.
+ * @param table Hoodie Table
+ * @param upsertHandle Merge Handle
+ * @param <T>
+ * @throws IOException in case of error
+ */
+ public static <T extends HoodieRecordPayload<T>> void runMerge(HoodieTable<T> table,
+ HoodieMergeHandle<T> upsertHandle) throws IOException {
+ final boolean externalchemaTransformation = table.getConfig().shouldUseExternalSchemaTransformation();
+ Configuration configForHudiFile = new Configuration(table.getHadoopConf());
+ HoodieBaseFile baseFile = upsertHandle.getPrevBaseFile();
+
+ final GenericDatumWriter<GenericRecord> gWriter;
+ final GenericDatumReader<GenericRecord> gReader;
+ if (externalchemaTransformation || baseFile.getExternalBaseFile().isPresent()) {
+ MessageType usedParquetSchema = ParquetUtils.readSchema(table.getHadoopConf(), upsertHandle.getOldFilePath());
Review comment:
For reference
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter edited a comment on pull request #1678: [WIP] [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#issuecomment-636276146
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=h1) Report
> Merging [#1678](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/3574a89232ad8573d1f76b27f127cc2ab9195e38&el=desc) will **decrease** coverage by `1.06%`.
> The diff coverage is `6.93%`.
[](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1678 +/- ##
============================================
- Coverage 18.22% 17.16% -1.07%
- Complexity 857 888 +31
============================================
Files 348 388 +40
Lines 15332 16758 +1426
Branches 1523 1644 +121
============================================
+ Hits 2794 2876 +82
- Misses 12181 13514 +1333
- Partials 357 368 +11
```
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...java/org/apache/hudi/client/HoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `18.77% <0.00%> (-1.61%)` | `12.00 <0.00> (ø)` | |
| [.../hudi/client/bootstrap/BootstrapRecordPayload.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Cb290c3RyYXBSZWNvcmRQYXlsb2FkLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...lient/bootstrap/BootstrapSourceSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Cb290c3RyYXBTb3VyY2VTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...he/hudi/client/bootstrap/BootstrapWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Cb290c3RyYXBXcml0ZVN0YXR1cy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...i/client/bootstrap/FullBootstrapInputProvider.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9GdWxsQm9vdHN0cmFwSW5wdXRQcm92aWRlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...e/hudi/client/bootstrap/HoodieBootstrapClient.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Ib29kaWVCb290c3RyYXBDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...lient/bootstrap/MetadataBootstrapKeyGenerator.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9NZXRhZGF0YUJvb3RzdHJhcEtleUdlbmVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...ient/bootstrap/selector/BootstrapModeSelector.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9zZWxlY3Rvci9Cb290c3RyYXBNb2RlU2VsZWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...bootstrap/selector/BootstrapRegexModeSelector.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9zZWxlY3Rvci9Cb290c3RyYXBSZWdleE1vZGVTZWxlY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [.../bootstrap/selector/FullBootstrapModeSelector.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9zZWxlY3Rvci9GdWxsQm9vdHN0cmFwTW9kZVNlbGVjdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| ... and [128 more](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=footer). Last update [3574a89...8824a93](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r439127629
##########
File path: .travis.yml
##########
@@ -13,14 +13,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+os: linux
language: java
dist: trusty
jdk:
- openjdk8
-sudo: required
-env:
- - HUDI_QUIETER_LOGGING=1 TEST_SUITE=unit
- - TEST_SUITE=integration
+jobs:
Review comment:
So `hudi-common` will be run for both?
##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/StatsCommand.java
##########
@@ -124,13 +125,15 @@ public String fileSizeStats(
Histogram globalHistogram = new Histogram(new UniformReservoir(MAX_FILES));
HashMap<String, Histogram> commitHistoMap = new HashMap<>();
for (FileStatus fileStatus : statuses) {
- String instantTime = FSUtils.getCommitTime(fileStatus.getPath().getName());
- long sz = fileStatus.getLen();
- if (!commitHistoMap.containsKey(instantTime)) {
- commitHistoMap.put(instantTime, new Histogram(new UniformReservoir(MAX_FILES)));
+ if (!fileStatus.getPath().toString().contains(HoodieTableMetaClient.METAFOLDER_NAME)) {
Review comment:
I wonder if such a listing should be moved inside a Utils methods in FSUtils and not guarded at the CLI level?
##########
File path: docker/demo/sparksql-batch2.commands
##########
@@ -26,4 +26,14 @@ spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from s
spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
+ // Copy-On-Write Bootstrapped table
Review comment:
wondering if we really need normal and bootstrapped tables are two cases here.. Is nt it not enough to just to cover at the partition level? i.e turn the one of the partitions in the the existing tables to be bootstrapped? that way we don't expand the existing test runtimes.
##########
File path: hudi-cli/pom.xml
##########
@@ -147,6 +147,43 @@
</execution>
</executions>
</plugin>
+<!--
Review comment:
if its going to be commented out, remove?
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/FullBootstrapInputProvider.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap;
+
+import java.io.Serializable;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.collection.Pair;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.util.List;
+
+/**
+ * Creates RDD of Hoodie Records given a list of partitions.
+ */
+public abstract class FullBootstrapInputProvider implements Serializable {
Review comment:
make an interface for this? that way we can even bootstrap data out of other table formats, by writing a simple class.
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/selector/BootstrapRegexModeSelector.java
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap.selector;
+
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+public class BootstrapRegexModeSelector extends BootstrapModeSelector {
+
+ private final Pattern pattern;
+ private final BootstrapMode bootstrapModeOnMatch;
+ private final BootstrapMode defaultMode;
+
+ public BootstrapRegexModeSelector(HoodieWriteConfig writeConfig) {
+ super(writeConfig);
+ this.pattern = Pattern.compile(writeConfig.getBootstrapModeSelectorRegex());
+ this.bootstrapModeOnMatch = writeConfig.getBootstrapModeForRegexMatch();
+ this.defaultMode = BootstrapMode.FULL_BOOTSTRAP.equals(bootstrapModeOnMatch)
Review comment:
this part is bit confusing.... so we take a regex to match, and the mode (full/metadata).. we apply the configured mode to matching partitions.. and the other mode to the rest? is it better to just make another flag with sensible defaults for both? I can foresee having to explain this to users..
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/selector/BootstrapModeSelector.java
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap.selector;
+
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Pluggable Partition Selector for selecting partitions to perform full or metadata-only bootstrapping.
+ */
+public abstract class BootstrapModeSelector implements Serializable {
+
+ protected final HoodieWriteConfig writeConfig;
+
+ public BootstrapModeSelector(HoodieWriteConfig writeConfig) {
+ this.writeConfig = writeConfig;
+ }
+
+ /**
+ * Classify partitions for the purpose of bootstrapping. For a non-partitioned source, input list will be one entry.
+ *
+ * @param partitions List of partitions with files present in each partitions
+ * @return a partitions grouped by bootstrap mode
+ */
+ public abstract Map<BootstrapMode, List<String>> select(
+ List<Pair<String, List<HoodieFileStatus>>> partitions);
+}
Review comment:
nit: new line
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -147,6 +147,35 @@ public static SparkConf registerClasses(SparkConf conf) {
return recordsWithLocation.filter(v1 -> !v1.isCurrentLocationKnown());
}
+ /**
+ * Main API to run bootstrap to hudi.
+ */
+ public void bootstrap(Option<Map<String, String>> extraMetadata) {
Review comment:
I would kind of expect the `bootStrapFromPath` to be passed in here? I guess that's part of the HoodieWriteConfig?
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -146,6 +146,35 @@ public static SparkConf registerClasses(SparkConf conf) {
return recordsWithLocation.filter(v1 -> !v1.isCurrentLocationKnown());
}
+ /**
+ * Main API to run bootstrap to hudi.
+ */
+ public void bootstrap(Option<Map<String, String>> extraMetadata) {
+ if (rollbackPending) {
+ rollBackPendingBootstrap();
+ }
+ HoodieTable<T> table = getTableAndInitCtx(WriteOperationType.UPSERT);
+ table.bootstrap(jsc, extraMetadata);
+ }
+
+ /**
+ * Main API to rollback pending bootstrap.
+ */
+ protected void rollBackPendingBootstrap() {
Review comment:
On the naming, I feel we should stick to either inflight, requested or complete i.e rollbackInflightBootstrap or rollbackIncompleteRollback(). (pending is a new term, we are introducing?) . may be this ship has already sailed
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/BootstrapMode.java
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap;
+
+public enum BootstrapMode {
Review comment:
javadocs
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/selector/BootstrapRegexModeSelector.java
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap.selector;
+
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+public class BootstrapRegexModeSelector extends BootstrapModeSelector {
+
+ private final Pattern pattern;
+ private final BootstrapMode bootstrapModeOnMatch;
+ private final BootstrapMode defaultMode;
+
+ public BootstrapRegexModeSelector(HoodieWriteConfig writeConfig) {
+ super(writeConfig);
+ this.pattern = Pattern.compile(writeConfig.getBootstrapModeSelectorRegex());
+ this.bootstrapModeOnMatch = writeConfig.getBootstrapModeForRegexMatch();
+ this.defaultMode = BootstrapMode.FULL_BOOTSTRAP.equals(bootstrapModeOnMatch)
+ ? BootstrapMode.METADATA_ONLY_BOOTSTRAP : BootstrapMode.FULL_BOOTSTRAP;
+ System.out.println("Default Mode :" + defaultMode + ", on Match Mode :" + bootstrapModeOnMatch);
Review comment:
logger?
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -146,6 +146,35 @@ public static SparkConf registerClasses(SparkConf conf) {
return recordsWithLocation.filter(v1 -> !v1.isCurrentLocationKnown());
}
+ /**
+ * Main API to run bootstrap to hudi.
+ */
+ public void bootstrap(Option<Map<String, String>> extraMetadata) {
+ if (rollbackPending) {
+ rollBackPendingBootstrap();
+ }
+ HoodieTable<T> table = getTableAndInitCtx(WriteOperationType.UPSERT);
+ table.bootstrap(jsc, extraMetadata);
+ }
+
+ /**
+ * Main API to rollback pending bootstrap.
+ */
+ protected void rollBackPendingBootstrap() {
Review comment:
for testing it still works if we drop the `protected` correct? keeping it package private
##########
File path: hudi-cli/pom.xml
##########
@@ -202,6 +237,12 @@
<type>test-jar</type>
</dependency>
+ <dependency>
Review comment:
should we remove other dependencies from this pom then? hudi-common, hudi-client etc will be inside the utilities bundle anyway
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/BootstrapMode.java
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap;
+
+public enum BootstrapMode {
+ FULL_BOOTSTRAP,
+ METADATA_ONLY_BOOTSTRAP
Review comment:
rename to
`RECORD_DATA_BOOTSTRAP`, `RECORD_METADATA_BOOTSTRAP` (or)
just `THICK_BOOTSTRAP`, `THIN_BOOTSTRAP` that more user understandable and relatable to eachother?
We atleast need to call it `RECORD_METADATA_ONLY_BOOTSTRAP` since now, it feels like we are just copying file metadata.. which we are not..
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/BootstrapSourceSchemaProvider.java
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap;
+
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import org.apache.avro.Schema;
+import org.apache.hadoop.fs.Path;
+
+import java.util.List;
+
+/**
+ * Bootstrap Schema Provider. Schema provided in config is used. If not available, use schema from Parquet
+ */
+public class BootstrapSourceSchemaProvider {
+
+ protected final HoodieWriteConfig bootstrapConfig;
Review comment:
rename to writeConfig, unless this only has bootstrap related properties
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/translator/MetadataBootstrapPartitionPathTranslator.java
##########
@@ -16,28 +16,23 @@
* limitations under the License.
*/
-package org.apache.hudi.keygen;
-
-import org.apache.hudi.common.config.TypedProperties;
-import org.apache.hudi.common.model.HoodieKey;
-
-import org.apache.avro.generic.GenericRecord;
+package org.apache.hudi.client.bootstrap.translator;
Review comment:
why is linkedin against the keygenerator file?
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/selector/MetadataOnlyBootstrapModeSelector.java
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap.selector;
+
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+public class MetadataOnlyBootstrapModeSelector extends UniBootstrapModeSelector {
Review comment:
we need to rename these classes nicely as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#issuecomment-640020465
I will look at any new test failures if it happens. It will likely be of log-limit or other CI specific issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r463295745
##########
File path: hudi-client/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieKeyException;
+
+public class KeyGenUtils {
Review comment:
I merged/rebased all this in #1876
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #1678: [WIP] [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r435985550
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/selector/BootstrapRegexModeSelector.java
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap.selector;
+
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+public class BootstrapRegexModeSelector extends BootstrapModeSelector {
Review comment:
For Mixed mode bootstrapping, this class can be used to select few partitions for full bootstrap while configuring metadata bootstrap for others.
##########
File path: .travis.yml
##########
@@ -13,14 +13,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+os: linux
language: java
dist: trusty
jdk:
- openjdk8
-sudo: required
-env:
- - HUDI_QUIETER_LOGGING=1 TEST_SUITE=unit
- - TEST_SUITE=integration
+jobs:
Review comment:
Seeing timeout to CI runs. Went into a hole trying to reuse Spark Context. Ended up splitting CI run for hudi-client and others. This is from #1619
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/BootstrapWriteStatus.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap;
+
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.util.collection.Pair;
+
+/**
+ * WriteStatus for Bootstrap.
+ */
+public class BootstrapWriteStatus extends WriteStatus {
Review comment:
Used for metadata bootstrap. Contains information needed to create bootstrap index.
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -146,6 +146,35 @@ public static SparkConf registerClasses(SparkConf conf) {
return recordsWithLocation.filter(v1 -> !v1.isCurrentLocationKnown());
}
+ /**
+ * Main API to run bootstrap to hudi.
+ */
+ public void bootstrap(Option<Map<String, String>> extraMetadata) {
+ if (rollbackPending) {
+ rollBackPendingBootstrap();
+ }
+ HoodieTable<T> table = getTableAndInitCtx(WriteOperationType.UPSERT);
+ table.bootstrap(jsc, extraMetadata);
+ }
+
+ /**
+ * Main API to rollback pending bootstrap.
+ */
+ protected void rollBackPendingBootstrap() {
Review comment:
internal method made visible for testing. Bootstrap rollback is special as it would have to revert 2 commits (when both metadata and full bootstrap is configured)
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieBootstrapConfig.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPathTranslator;
+import org.apache.hudi.common.config.DefaultHoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Bootstrap specific configs.
+ */
+public class HoodieBootstrapConfig extends DefaultHoodieConfig {
+
+ public static final String SOURCE_BASE_PATH_PROP = "hoodie.bootstrap.source.base.path";
Review comment:
External Source Path
##########
File path: hudi-client/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
Review comment:
Moved base class to hudi-client as this is needed for metadata bootstrap
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/translator/MetadataBootstrapPartitionPathTranslator.java
##########
@@ -16,28 +16,23 @@
* limitations under the License.
*/
-package org.apache.hudi.keygen;
-
-import org.apache.hudi.common.config.TypedProperties;
-import org.apache.hudi.common.model.HoodieKey;
-
-import org.apache.avro.generic.GenericRecord;
+package org.apache.hudi.client.bootstrap.translator;
import java.io.Serializable;
+import org.apache.hudi.common.config.TypedProperties;
-/**
- * Abstract class to extend for plugging in extraction of {@link HoodieKey} from an Avro record.
- */
-public abstract class KeyGenerator implements Serializable {
+public abstract class MetadataBootstrapPartitionPathTranslator implements Serializable {
Review comment:
HUDI-1001 to add implementations
##########
File path: hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieDemo.java
##########
@@ -52,11 +53,18 @@
private static final String COW_TABLE_NAME = "stock_ticks_cow";
private static final String MOR_TABLE_NAME = "stock_ticks_mor";
+ private static final String BOOTSTRAPPED_SRC_PATH = "/user/hive/warehouse/stock_ticks_cow_bs_src";
Review comment:
what we do here : Once the first of ingestion is completed for COW and MOR table, we create a new parquet dataset by reading from COW. We then use this new parquet dataset to perform metadata bootstrap onto a new COW and MOR table. Hive and Spark SQL queries are run against these tables. We also perform upserts and compaction (MOR) on these metadata bootstrap tables and test through hive and spark-sql. Presto needs to be added after we support it.
##########
File path: packaging/hudi-hadoop-mr-bundle/pom.xml
##########
@@ -72,9 +72,19 @@
<include>org.objenesis:objenesis</include>
<include>com.esotericsoftware:minlog</include>
<include>org.apache.avro:avro</include>
+ <include>org.apache.hbase:hbase-common</include>
Review comment:
we need to add all these to bundles in order include HFile at runtime across query engines (fyi : @prashantwason )
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
##########
@@ -122,20 +145,37 @@ public void shutdownGracefully() {
* @throws Exception
*/
public void sync() throws Exception {
- if (cfg.continuousMode) {
- deltaSyncService.start(this::onDeltaSyncShutdown);
- deltaSyncService.waitForShutdown();
- LOG.info("Delta Sync shutting down");
+ if (bootstrapExecutor.isPresent()) {
Review comment:
Users are expected to run bootstrap executor in a stand-alone mode. Then they can switch to continuous mode for subsequent ingestion.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java
##########
@@ -81,6 +81,13 @@
String INVALID_INSTANT_TS = "0";
+ // Instant corresponding to pristine state of the table after its creation
Review comment:
I am using a value higher than INIT_INSTANT_TS for bootstrap instants so that we will have uniform semantics for incrementally reading bootstrap vs non-bootstrap commits.
##########
File path: hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestRepairsCommand.java
##########
@@ -164,7 +164,7 @@ public void testOverwriteHoodieProperties() throws IOException {
// check result
List<String> allPropsStr = Arrays.asList("hoodie.table.name", "hoodie.table.type",
- "hoodie.archivelog.folder", "hoodie.timeline.layout.version");
+ "hoodie.archivelog.folder", "hoodie.bootstrap.index.class", "hoodie.timeline.layout.version");
Review comment:
Bootstrap Index implementation is pluggable per request from AWS.
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieBootstrapConfig.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPathTranslator;
+import org.apache.hudi.common.config.DefaultHoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Bootstrap specific configs.
+ */
+public class HoodieBootstrapConfig extends DefaultHoodieConfig {
+
+ public static final String SOURCE_BASE_PATH_PROP = "hoodie.bootstrap.source.base.path";
+ public static final String BOOTSTRAP_MODE_SELECTOR = "hoodie.bootstrap.mode.selector";
+ public static final String FULL_BOOTRAP_INPUT_PROVIDER = "hoodie.bootstrap.full.input.provider";
+ public static final String BOOTSTRAP_KEYGEN_CLASS = "hoodie.bootstrap.keygen.class";
Review comment:
Note: Same keygen class needs to be configured as the one used for regular upserts to have a consistent upsert behavior after bootstrap.
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/utils/MergingParquetIterator.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import java.util.Iterator;
+import java.util.function.Function;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hudi.common.util.collection.Pair;
+
+public class MergingParquetIterator<T extends GenericRecord> implements Iterator<T> {
Review comment:
Used to provide stitched records from older file slice for upsert/compact on metadata bootstrapped table.
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java
##########
@@ -173,23 +173,26 @@ protected void updateIndexAndCommitIfNeeded(JavaRDD<WriteStatus> writeStatusRDD,
protected void commitOnAutoCommit(HoodieWriteMetadata result) {
if (config.shouldAutoCommit()) {
LOG.info("Auto commit enabled: Committing " + instantTime);
- commit(Option.empty(), result);
+ commit(extraMetadata, result);
} else {
LOG.info("Auto commit disabled for " + instantTime);
}
}
- private void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
Review comment:
Metadata bootstrap needs Write Statuses for creating bootstrap index. But, we also need to ensure collect on the Write Status RDD is done only once. Hence, this change.
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieBootstrapConfig.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPathTranslator;
+import org.apache.hudi.common.config.DefaultHoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Bootstrap specific configs.
+ */
+public class HoodieBootstrapConfig extends DefaultHoodieConfig {
+
+ public static final String SOURCE_BASE_PATH_PROP = "hoodie.bootstrap.source.base.path";
+ public static final String BOOTSTRAP_MODE_SELECTOR = "hoodie.bootstrap.mode.selector";
Review comment:
Configure class which decides partitions for doing metadata/full bootstrap.
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -118,14 +119,17 @@
"_.hoodie.allow.multi.write.on.same.instant";
private static final String DEFAULT_ALLOW_MULTI_WRITE_ON_SAME_INSTANT = "false";
+ public static final String EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION = AVRO_SCHEMA + ".externalTransformation";
Review comment:
Parquet-avro has a bug when dealing with nested fields which are lists. It looks like Parquet-avro changes the schema field name of this nested field.
Schema used to write : https://gist.github.com/bvaradar/fb7af49258b8bfc232607a06d3adc8db#file-writer-schema-nested-array-schema-L70
Schema returned by Parquet-Avro : https://gist.github.com/bvaradar/c69d9af4cc7b62bf81e4cf938d0ba9ca#file-same-schema-when-read-from-parquet-L72
Using explicit record translation to new schema in Hudi gets it to work. Logic in MergeHelper.java
##########
File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieBootstrapHandle.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+public class HoodieBootstrapHandle<T extends HoodieRecordPayload> extends HoodieCreateHandle<T> {
Review comment:
Used by metadata bootstrap to ensure the schema is set correctly and skeleton and external files is 1-1.
##########
File path: hudi-client/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieKeyException;
+
+public class KeyGenUtils {
Review comment:
Most KeyGenerator classes have boiler plate code. Have refactored to ensure they are handle them consistently. Also, I needed separate APIs to get only the record key (not partition-path) for Metadata Bootstrap. Made changes in KeyGenerator classes for that.
##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/StatsCommand.java
##########
@@ -124,13 +125,15 @@ public String fileSizeStats(
Histogram globalHistogram = new Histogram(new UniformReservoir(MAX_FILES));
HashMap<String, Histogram> commitHistoMap = new HashMap<>();
for (FileStatus fileStatus : statuses) {
- String instantTime = FSUtils.getCommitTime(fileStatus.getPath().getName());
- long sz = fileStatus.getLen();
- if (!commitHistoMap.containsKey(instantTime)) {
- commitHistoMap.put(instantTime, new Histogram(new UniformReservoir(MAX_FILES)));
+ if (!fileStatus.getPath().toString().contains(HoodieTableMetaClient.METAFOLDER_NAME)) {
Review comment:
CLI Tests caught this issue !! .bootstrap folder is under .hoodie which shows up in this listing
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/HoodieCopyOnWriteTable.java
##########
@@ -155,23 +161,10 @@ public HoodieWriteMetadata compact(JavaSparkContext jsc, String compactionInstan
throw new HoodieUpsertException(
"Error in finding the old file path at commit " + instantTime + " for fileId: " + fileId);
} else {
- AvroReadSupport.setAvroReadSchema(getHadoopConf(), upsertHandle.getWriterSchema());
- BoundedInMemoryExecutor<GenericRecord, GenericRecord, Void> wrapper = null;
- try (ParquetReader<IndexedRecord> reader =
- AvroParquetReader.<IndexedRecord>builder(upsertHandle.getOldFilePath()).withConf(getHadoopConf()).build()) {
- wrapper = new SparkBoundedInMemoryExecutor(config, new ParquetReaderIterator(reader),
- new UpdateHandler(upsertHandle), x -> x);
- wrapper.execute();
- } catch (Exception e) {
- throw new HoodieException(e);
- } finally {
- upsertHandle.close();
- if (null != wrapper) {
- wrapper.shutdownNow();
- }
- }
+ MergeHelper.runMerge(this, upsertHandle);
Review comment:
Moved to avoid duplication in handling merge for both compaction and COW upsert
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieBootstrapConfig.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPathTranslator;
+import org.apache.hudi.common.config.DefaultHoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Bootstrap specific configs.
+ */
+public class HoodieBootstrapConfig extends DefaultHoodieConfig {
+
+ public static final String SOURCE_BASE_PATH_PROP = "hoodie.bootstrap.source.base.path";
+ public static final String BOOTSTRAP_MODE_SELECTOR = "hoodie.bootstrap.mode.selector";
+ public static final String FULL_BOOTRAP_INPUT_PROVIDER = "hoodie.bootstrap.full.input.provider";
Review comment:
Configure class which provides RDD given a list of partitions. Defaults are provided at DeltaStreamer/Spark-DataSource level.
@vinothchandar : This doesnt have to be configurable but it is kept this way as a dependency injection to avoid moving SparkDataSourceBasedFullBootstrapInputProvider and all its dependencies from hudi-spark to hudi-client. Would need a discussion to come up with cleaner approach.
##########
File path: hudi-cli/pom.xml
##########
@@ -202,6 +237,12 @@
<type>test-jar</type>
</dependency>
+ <dependency>
Review comment:
Hudi-utilities-bundle shades hbase classes which also ends up in bootstrap file itself (key class is added in footer) and hence we need to use bundle even in CLI when doing any operations involving bootstrap table.
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,532 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.HashSet;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.MetadataBootstrapKeyGenerator;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.TableNotFoundException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, MetadataBootstrapKeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
+ ParquetMetadataConverter.NO_FILTER);
+ MessageType parquetSchema = readFooter.getFileMetaData().getSchema();
+ avroSchema = new AvroSchemaConverter().convert(parquetSchema);
+ Schema recordKeySchema = HoodieAvroUtils.generateProjectionSchema(avroSchema,
+ keyGenerator.getTopLevelRecordKeyFields());
+ LOG.info("Schema to be used for reading record Keys :" + recordKeySchema);
+ AvroReadSupport.setAvroReadSchema(table.getHadoopConf(), recordKeySchema);
+ AvroReadSupport.setRequestedProjection(table.getHadoopConf(), recordKeySchema);
+
+ BoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void> wrapper = null;
+ try (ParquetReader<IndexedRecord> reader =
+ AvroParquetReader.<IndexedRecord>builder(sourceFilePath).withConf(table.getHadoopConf()).build()) {
+ wrapper = new SparkBoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void>(config,
+ new ParquetReaderIterator(reader), new BootstrapRecordWriter(bootstrapHandle), inp -> {
+ String recKey = keyGenerator.getRecordKey(inp);
+ GenericRecord gr = new GenericData.Record(HoodieAvroUtils.RECORD_KEY_SCHEMA);
+ gr.put(HoodieRecord.RECORD_KEY_METADATA_FIELD, recKey);
+ BootstrapRecordPayload payload = new BootstrapRecordPayload(gr);
+ HoodieRecord rec = new HoodieRecord(new HoodieKey(recKey, partitionPath), payload);
+ return rec;
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ bootstrapHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ BootstrapWriteStatus writeStatus = (BootstrapWriteStatus)bootstrapHandle.getWriteStatus();
+ BootstrapSourceFileMapping bootstrapSourceFileMapping = new BootstrapSourceFileMapping(
+ config.getBootstrapSourceBasePath(), srcPartitionPath, partitionPath,
+ srcFileStatus, writeStatus.getFileId());
+ writeStatus.setBootstrapSourceFileMapping(bootstrapSourceFileMapping);
+ return writeStatus;
+ }
+
+ /**
+ * Return Bootstrap Mode selections for partitions listed and figure out bootstrap Schema.
+ * @param metaClient Hoodie Table Meta Client.
+ * @return
+ * @throws IOException
+ */
+ private Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> listAndProcessSourcePartitions(
+ HoodieTableMetaClient metaClient) throws IOException {
+ //TODO: Added HoodieFilter for manually testing bootstrap from source hudi table. Needs to be reverted.
+ final PathFilter hoodieFilter = new HoodieROTablePathFilter();
+ List<Pair<String, List<HoodieFileStatus>>> folders =
Review comment:
HUDI-999 to track parallelizing this step.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/FileStatusDTO.java
##########
@@ -68,7 +68,7 @@ public static FileStatusDTO fromFileStatus(FileStatus fileStatus) {
dto.blockReplication = fileStatus.getReplication();
dto.blocksize = fileStatus.getBlockSize();
dto.modificationTime = fileStatus.getModificationTime();
- dto.accessTime = fileStatus.getModificationTime();
+ dto.accessTime = fileStatus.getAccessTime();
Review comment:
FYI : this is a bug but doesn't have any impact currently.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r463409859
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/selector/BootstrapRegexModeSelector.java
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap.selector;
+
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+public class BootstrapRegexModeSelector extends BootstrapModeSelector {
+
+ private final Pattern pattern;
+ private final BootstrapMode bootstrapModeOnMatch;
+ private final BootstrapMode defaultMode;
+
+ public BootstrapRegexModeSelector(HoodieWriteConfig writeConfig) {
+ super(writeConfig);
+ this.pattern = Pattern.compile(writeConfig.getBootstrapModeSelectorRegex());
+ this.bootstrapModeOnMatch = writeConfig.getBootstrapModeForRegexMatch();
+ this.defaultMode = BootstrapMode.FULL_BOOTSTRAP.equals(bootstrapModeOnMatch)
Review comment:
I am leaving this as-is even in #1876 as this need more user input and discusion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-commenter commented on pull request #1678: [WIP] [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#issuecomment-636276146
# [Codecov](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=h1) Report
> Merging [#1678](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=desc) into [master](https://codecov.io/gh/apache/hudi/commit/3574a89232ad8573d1f76b27f127cc2ab9195e38&el=desc) will **decrease** coverage by `1.06%`.
> The diff coverage is `6.93%`.
[](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #1678 +/- ##
============================================
- Coverage 18.22% 17.16% -1.07%
- Complexity 857 888 +31
============================================
Files 348 388 +40
Lines 15332 16758 +1426
Branches 1523 1644 +121
============================================
+ Hits 2794 2876 +82
- Misses 12181 13514 +1333
- Partials 357 368 +11
```
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [...java/org/apache/hudi/client/HoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `18.77% <0.00%> (-1.61%)` | `12.00 <0.00> (ø)` | |
| [.../hudi/client/bootstrap/BootstrapRecordPayload.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Cb290c3RyYXBSZWNvcmRQYXlsb2FkLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...lient/bootstrap/BootstrapSourceSchemaProvider.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Cb290c3RyYXBTb3VyY2VTY2hlbWFQcm92aWRlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...he/hudi/client/bootstrap/BootstrapWriteStatus.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Cb290c3RyYXBXcml0ZVN0YXR1cy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...i/client/bootstrap/FullBootstrapInputProvider.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9GdWxsQm9vdHN0cmFwSW5wdXRQcm92aWRlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...e/hudi/client/bootstrap/HoodieBootstrapClient.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9Ib29kaWVCb290c3RyYXBDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...lient/bootstrap/MetadataBootstrapKeyGenerator.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9NZXRhZGF0YUJvb3RzdHJhcEtleUdlbmVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...ient/bootstrap/selector/BootstrapModeSelector.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9zZWxlY3Rvci9Cb290c3RyYXBNb2RlU2VsZWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...bootstrap/selector/BootstrapRegexModeSelector.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9zZWxlY3Rvci9Cb290c3RyYXBSZWdleE1vZGVTZWxlY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [.../bootstrap/selector/FullBootstrapModeSelector.java](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree#diff-aHVkaS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2Jvb3RzdHJhcC9zZWxlY3Rvci9GdWxsQm9vdHN0cmFwTW9kZVNlbGVjdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| ... and [128 more](https://codecov.io/gh/apache/hudi/pull/1678/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=footer). Last update [3574a89...8824a93](https://codecov.io/gh/apache/hudi/pull/1678?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r444210548
##########
File path: hudi-cli/pom.xml
##########
@@ -147,6 +147,43 @@
</execution>
</executions>
</plugin>
+<!--
Review comment:
done
##########
File path: hudi-cli/pom.xml
##########
@@ -202,6 +237,12 @@
<type>test-jar</type>
</dependency>
+ <dependency>
Review comment:
done
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieBootstrapConfig.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPathTranslator;
+import org.apache.hudi.common.config.DefaultHoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Bootstrap specific configs.
+ */
+public class HoodieBootstrapConfig extends DefaultHoodieConfig {
+
+ public static final String SOURCE_BASE_PATH_PROP = "hoodie.bootstrap.source.base.path";
Review comment:
Done
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/bootstrap/BootstrapMode.java
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.bootstrap;
+
+public enum BootstrapMode {
+ FULL_BOOTSTRAP,
+ METADATA_ONLY_BOOTSTRAP
Review comment:
Done
##########
File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
##########
@@ -59,18 +60,31 @@
public HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath,
String fileId, HoodieTable<T> hoodieTable, SparkTaskContextSupplier sparkTaskContextSupplier) {
+ this(config, instantTime, partitionPath, fileId, hoodieTable, generateOriginalAndHoodieWriteSchema(config),
+ sparkTaskContextSupplier);
+ }
+
+ protected HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath, String fileId,
+ HoodieTable<T> hoodieTable, Pair<Schema, Schema> originalAndHoodieSchema,
+ SparkTaskContextSupplier sparkTaskContextSupplier) {
super(config, instantTime, hoodieTable);
this.partitionPath = partitionPath;
this.fileId = fileId;
- this.originalSchema = new Schema.Parser().parse(config.getSchema());
- this.writerSchema = HoodieAvroUtils.createHoodieWriteSchema(originalSchema);
+ this.originalSchema = originalAndHoodieSchema.getKey();
+ this.writerSchema = originalAndHoodieSchema.getValue();
this.timer = new HoodieTimer().startTimer();
this.writeStatus = (WriteStatus) ReflectionUtils.loadClass(config.getWriteStatusClassName(),
!hoodieTable.getIndex().isImplicitWithStorage(), config.getWriteStatusFailureFraction());
this.sparkTaskContextSupplier = sparkTaskContextSupplier;
this.writeToken = makeWriteToken();
}
+ protected static Pair<Schema, Schema> generateOriginalAndHoodieWriteSchema(HoodieWriteConfig config) {
+ Schema originalSchema = new Schema.Parser().parse(config.getSchema());
+ Schema hoodieSchema = HoodieAvroUtils.addMetadataFields(originalSchema);
Review comment:
Done
##########
File path: .travis.yml
##########
@@ -13,14 +13,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+os: linux
language: java
dist: trusty
jdk:
- openjdk8
-sudo: required
-env:
- - HUDI_QUIETER_LOGGING=1 TEST_SUITE=unit
- - TEST_SUITE=integration
+jobs:
Review comment:
hudi-common tests are run only in one of the runs
##########
File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java
##########
@@ -57,7 +59,14 @@
public HoodieCreateHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T> hoodieTable,
String partitionPath, String fileId, SparkTaskContextSupplier sparkTaskContextSupplier) {
- super(config, instantTime, partitionPath, fileId, hoodieTable, sparkTaskContextSupplier);
+ this(config, instantTime, hoodieTable, partitionPath, fileId, generateOriginalAndHoodieWriteSchema(config),
Review comment:
Renamed as per suggestion below
##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/StatsCommand.java
##########
@@ -124,13 +125,15 @@ public String fileSizeStats(
Histogram globalHistogram = new Histogram(new UniformReservoir(MAX_FILES));
HashMap<String, Histogram> commitHistoMap = new HashMap<>();
for (FileStatus fileStatus : statuses) {
- String instantTime = FSUtils.getCommitTime(fileStatus.getPath().getName());
- long sz = fileStatus.getLen();
- if (!commitHistoMap.containsKey(instantTime)) {
- commitHistoMap.put(instantTime, new Histogram(new UniformReservoir(MAX_FILES)));
+ if (!fileStatus.getPath().toString().contains(HoodieTableMetaClient.METAFOLDER_NAME)) {
Review comment:
Added a method in FSUtils for this and reused it in other places also
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -146,6 +146,35 @@ public static SparkConf registerClasses(SparkConf conf) {
return recordsWithLocation.filter(v1 -> !v1.isCurrentLocationKnown());
}
+ /**
+ * Main API to run bootstrap to hudi.
+ */
+ public void bootstrap(Option<Map<String, String>> extraMetadata) {
+ if (rollbackPending) {
+ rollBackPendingBootstrap();
+ }
+ HoodieTable<T> table = getTableAndInitCtx(WriteOperationType.UPSERT);
+ table.bootstrap(jsc, extraMetadata);
+ }
+
+ /**
+ * Main API to rollback pending bootstrap.
+ */
+ protected void rollBackPendingBootstrap() {
Review comment:
We are using the term Pending in Timeline for compactions. But, let me change to Inflight.
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/HoodieWriteClient.java
##########
@@ -147,6 +147,35 @@ public static SparkConf registerClasses(SparkConf conf) {
return recordsWithLocation.filter(v1 -> !v1.isCurrentLocationKnown());
}
+ /**
+ * Main API to run bootstrap to hudi.
+ */
+ public void bootstrap(Option<Map<String, String>> extraMetadata) {
Review comment:
Yes, its part of write config.
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
Review comment:
@umehrot2 : Did the setup where bootstrap and source path in different S3 buckets work end to end ? I remember you were mentioning about this testing.
##########
File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieBootstrapHandle.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+public class HoodieBootstrapHandle<T extends HoodieRecordPayload> extends HoodieCreateHandle<T> {
Review comment:
Added comments
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
Review comment:
this is covered as part of TestBootstrap
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -835,18 +892,25 @@ public HoodieWriteConfig build() {
setDefaultOnCondition(props, !isCompactionConfigSet,
HoodieCompactionConfig.newBuilder().fromProperties(props).build());
setDefaultOnCondition(props, !isMetricsConfigSet, HoodieMetricsConfig.newBuilder().fromProperties(props).build());
+ setDefaultOnCondition(props, !isBootstrapConfigSet,
+ HoodieBootstrapConfig.newBuilder().fromProperties(props).build());
setDefaultOnCondition(props, !isMemoryConfigSet, HoodieMemoryConfig.newBuilder().fromProperties(props).build());
setDefaultOnCondition(props, !isViewConfigSet,
FileSystemViewStorageConfig.newBuilder().fromProperties(props).build());
setDefaultOnCondition(props, !isConsistencyGuardSet,
ConsistencyGuardConfig.newBuilder().fromProperties(props).build());
+ setDefaultOnCondition(props, !props.containsKey(EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION),
+ EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION, DEFAULT_EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION);
setDefaultOnCondition(props, !props.containsKey(TIMELINE_LAYOUT_VERSION), TIMELINE_LAYOUT_VERSION,
String.valueOf(TimelineLayoutVersion.CURR_VERSION));
String layoutVersion = props.getProperty(TIMELINE_LAYOUT_VERSION);
// Ensure Layout Version is good
new TimelineLayoutVersion(Integer.parseInt(layoutVersion));
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r463405918
##########
File path: docker/demo/sparksql-batch2.commands
##########
@@ -26,4 +26,14 @@ spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from s
spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
+ // Copy-On-Write Bootstrapped table
Review comment:
working through these tests, its actually okay to have both of them separate for sometime, until bootstrap stabilizes well. all good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#issuecomment-670393921
closing this. There is a followup JIRA assigned to you @bvaradar with some of the unaddressed comments from here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r463990064
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/model/ExternalBaseFileMapping.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.model;
+
+import java.io.Serializable;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+
+/**
+ * POJO storing (partitionPath, hoodieFileId) -> external data file path.
+ */
+public class ExternalBaseFileMapping implements Serializable {
Review comment:
there is one method that's specific to base files. hmmm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#issuecomment-640020195
@vinothchandar : Added comments in the PR. Please take a look when you get a chance.
cc @umehrot2
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r463991884
##########
File path: hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/FileStatusWithExternalBaseFile.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hadoop;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+
+/**
+ * Sub-Type of File Status tracking both skeleton and external base file's status.
Review comment:
I did a pass. I ll leave this open for later
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r463286465
##########
File path: .travis.yml
##########
@@ -13,14 +13,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+os: linux
language: java
dist: trusty
jdk:
- openjdk8
-sudo: required
-env:
- - HUDI_QUIETER_LOGGING=1 TEST_SUITE=unit
- - TEST_SUITE=integration
+jobs:
Review comment:
I have reverted all of this now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r439521898
##########
File path: hudi-client/src/main/java/org/apache/hudi/keygen/KeyGenUtils.java
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieKeyException;
+
+public class KeyGenUtils {
Review comment:
this is similar to the some work pratyaksh was doing as well.. We need to probably reconcile these at some point
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieBootstrapConfig.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPathTranslator;
+import org.apache.hudi.common.config.DefaultHoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Bootstrap specific configs.
+ */
+public class HoodieBootstrapConfig extends DefaultHoodieConfig {
+
+ public static final String SOURCE_BASE_PATH_PROP = "hoodie.bootstrap.source.base.path";
Review comment:
Do we need the ‘.source’? Just call this bootstrap.basepath?
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -835,18 +892,25 @@ public HoodieWriteConfig build() {
setDefaultOnCondition(props, !isCompactionConfigSet,
HoodieCompactionConfig.newBuilder().fromProperties(props).build());
setDefaultOnCondition(props, !isMetricsConfigSet, HoodieMetricsConfig.newBuilder().fromProperties(props).build());
+ setDefaultOnCondition(props, !isBootstrapConfigSet,
+ HoodieBootstrapConfig.newBuilder().fromProperties(props).build());
setDefaultOnCondition(props, !isMemoryConfigSet, HoodieMemoryConfig.newBuilder().fromProperties(props).build());
setDefaultOnCondition(props, !isViewConfigSet,
FileSystemViewStorageConfig.newBuilder().fromProperties(props).build());
setDefaultOnCondition(props, !isConsistencyGuardSet,
ConsistencyGuardConfig.newBuilder().fromProperties(props).build());
+ setDefaultOnCondition(props, !props.containsKey(EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION),
+ EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION, DEFAULT_EXTERNAL_RECORD_AND_SCHEMA_TRANSFORMATION);
setDefaultOnCondition(props, !props.containsKey(TIMELINE_LAYOUT_VERSION), TIMELINE_LAYOUT_VERSION,
String.valueOf(TimelineLayoutVersion.CURR_VERSION));
String layoutVersion = props.getProperty(TIMELINE_LAYOUT_VERSION);
// Ensure Layout Version is good
new TimelineLayoutVersion(Integer.parseInt(layoutVersion));
Review comment:
should probably also break validation into its own method?
##########
File path: hudi-client/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieKey;
+
+import org.apache.avro.generic.GenericRecord;
+
+import java.io.Serializable;
+import org.apache.hudi.exception.HoodieKeyException;
+
+/**
+ * Abstract class to extend for plugging in extraction of {@link HoodieKey} from an Avro record.
+ */
+public abstract class KeyGenerator implements Serializable {
+
+ protected transient TypedProperties config;
+
+ protected KeyGenerator(TypedProperties config) {
+ this.config = config;
+ }
+
+ /**
+ * Generate a record Key out of provided generic record.
+ */
+ public abstract String getRecordKey(GenericRecord record);
+
+ /**
+ * Generate a partition path out of provided generic record.
+ */
+ public abstract String getPartitionPath(GenericRecord record);
+
+ /**
+ * Generate a Hoodie Key out of provided generic record.
+ */
+ public final HoodieKey getKey(GenericRecord record) {
+ if (getRecordKeyFields() == null || getPartitionPathFields() == null) {
+ throw new HoodieKeyException("Unable to find field names for record key or partition path in cfg");
+ }
+ return new HoodieKey(getRecordKey(record), getPartitionPath(record));
+ }
+
+ /**
+ * Used by Metadata bootstrap.
+ * @return
+ */
+ public final List<String> getTopLevelRecordKeyFields() {
+ // For nested columns, pick top level column name
+ return getRecordKeyFields().stream().map(k -> {
+ int idx = k.indexOf('.');
+ return idx > 0 ? k.substring(0, idx) : k;
+ }).collect(Collectors.toList());
+ }
+
+ /**
+ * Return fields that constitute record key. Used by Metadata bootstrap.
+ * Have a base implementation inorder to prevent forcing custom KeyGenerator implementation
+ * to implement this method
+ * @return list of record key fields
+ */
+ public List<String> getRecordKeyFields() {
Review comment:
so the custom KeyGenerators out there will break for bootstrap until they implement these methods? its best to avoid that
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieBootstrapConfig.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPathTranslator;
+import org.apache.hudi.common.config.DefaultHoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Bootstrap specific configs.
+ */
+public class HoodieBootstrapConfig extends DefaultHoodieConfig {
+
+ public static final String SOURCE_BASE_PATH_PROP = "hoodie.bootstrap.source.base.path";
+ public static final String BOOTSTRAP_MODE_SELECTOR = "hoodie.bootstrap.mode.selector";
+ public static final String FULL_BOOTRAP_INPUT_PROVIDER = "hoodie.bootstrap.full.input.provider";
+ public static final String BOOTSTRAP_KEYGEN_CLASS = "hoodie.bootstrap.keygen.class";
Review comment:
Then, why not re-use the keygen supplied for regular upsert?
##########
File path: hudi-client/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieKey;
+
+import org.apache.avro.generic.GenericRecord;
+
+import java.io.Serializable;
+import org.apache.hudi.exception.HoodieKeyException;
+
+/**
+ * Abstract class to extend for plugging in extraction of {@link HoodieKey} from an Avro record.
+ */
+public abstract class KeyGenerator implements Serializable {
+
+ protected transient TypedProperties config;
+
+ protected KeyGenerator(TypedProperties config) {
+ this.config = config;
+ }
+
+ /**
+ * Generate a record Key out of provided generic record.
+ */
+ public abstract String getRecordKey(GenericRecord record);
+
+ /**
+ * Generate a partition path out of provided generic record.
+ */
+ public abstract String getPartitionPath(GenericRecord record);
+
+ /**
+ * Generate a Hoodie Key out of provided generic record.
+ */
+ public final HoodieKey getKey(GenericRecord record) {
+ if (getRecordKeyFields() == null || getPartitionPathFields() == null) {
+ throw new HoodieKeyException("Unable to find field names for record key or partition path in cfg");
+ }
+ return new HoodieKey(getRecordKey(record), getPartitionPath(record));
+ }
+
+ /**
+ * Used by Metadata bootstrap.
+ * @return
+ */
+ public final List<String> getTopLevelRecordKeyFields() {
Review comment:
lets move this to some place else? better to avoid such feature specific methods in a external facing class
##########
File path: hudi-client/src/main/java/org/apache/hudi/keygen/KeyGenerator.java
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.model.HoodieKey;
+
+import org.apache.avro.generic.GenericRecord;
+
+import java.io.Serializable;
+import org.apache.hudi.exception.HoodieKeyException;
+
+/**
+ * Abstract class to extend for plugging in extraction of {@link HoodieKey} from an Avro record.
+ */
+public abstract class KeyGenerator implements Serializable {
+
+ protected transient TypedProperties config;
+
+ protected KeyGenerator(TypedProperties config) {
+ this.config = config;
+ }
+
+ /**
+ * Generate a record Key out of provided generic record.
+ */
+ public abstract String getRecordKey(GenericRecord record);
Review comment:
this is a problem.. it will break every KeyGenerator out there.. because they don't implement these methods.. we cannot have these two methods be `abstract`.. Why can we just get teh key/pp after calling getKey()
##########
File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
##########
@@ -59,18 +60,31 @@
public HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath,
String fileId, HoodieTable<T> hoodieTable, SparkTaskContextSupplier sparkTaskContextSupplier) {
+ this(config, instantTime, partitionPath, fileId, hoodieTable, generateOriginalAndHoodieWriteSchema(config),
+ sparkTaskContextSupplier);
+ }
+
+ protected HoodieWriteHandle(HoodieWriteConfig config, String instantTime, String partitionPath, String fileId,
+ HoodieTable<T> hoodieTable, Pair<Schema, Schema> originalAndHoodieSchema,
+ SparkTaskContextSupplier sparkTaskContextSupplier) {
super(config, instantTime, hoodieTable);
this.partitionPath = partitionPath;
this.fileId = fileId;
- this.originalSchema = new Schema.Parser().parse(config.getSchema());
- this.writerSchema = HoodieAvroUtils.createHoodieWriteSchema(originalSchema);
+ this.originalSchema = originalAndHoodieSchema.getKey();
+ this.writerSchema = originalAndHoodieSchema.getValue();
this.timer = new HoodieTimer().startTimer();
this.writeStatus = (WriteStatus) ReflectionUtils.loadClass(config.getWriteStatusClassName(),
!hoodieTable.getIndex().isImplicitWithStorage(), config.getWriteStatusFailureFraction());
this.sparkTaskContextSupplier = sparkTaskContextSupplier;
this.writeToken = makeWriteToken();
}
+ protected static Pair<Schema, Schema> generateOriginalAndHoodieWriteSchema(HoodieWriteConfig config) {
+ Schema originalSchema = new Schema.Parser().parse(config.getSchema());
+ Schema hoodieSchema = HoodieAvroUtils.addMetadataFields(originalSchema);
Review comment:
why don't we just call these : writerSchema , writeSchemaWithMetaFields?
##########
File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieBootstrapHandle.java
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.io;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.SparkTaskContextSupplier;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+public class HoodieBootstrapHandle<T extends HoodieRecordPayload> extends HoodieCreateHandle<T> {
Review comment:
its not obvious looking at this class, what it does.. may be more comments? for e,g why have `canWrite()` always return true
##########
File path: hudi-client/src/main/java/org/apache/hudi/io/HoodieCreateHandle.java
##########
@@ -57,7 +59,14 @@
public HoodieCreateHandle(HoodieWriteConfig config, String instantTime, HoodieTable<T> hoodieTable,
String partitionPath, String fileId, SparkTaskContextSupplier sparkTaskContextSupplier) {
- super(config, instantTime, partitionPath, fileId, hoodieTable, sparkTaskContextSupplier);
+ this(config, instantTime, hoodieTable, partitionPath, fileId, generateOriginalAndHoodieWriteSchema(config),
Review comment:
if possible, can we rename `original` to be something more meaningful.. it does not really tell me what that schema is.. is it the schema the prev version of the file was written with
##########
File path: hudi-client/src/main/java/org/apache/hudi/client/utils/MergingParquetIterator.java
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.client.utils;
+
+import java.util.Iterator;
+import java.util.function.Function;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hudi.common.util.collection.Pair;
+
+public class MergingParquetIterator<T extends GenericRecord> implements Iterator<T> {
+
+ private final ParquetReaderIterator<T> leftFileReader;
+ private final ParquetReaderIterator<T> rightFileReader;
+ private final Function<Pair<T,T>, T> mergeFunction;
+
+ public MergingParquetIterator(ParquetReaderIterator<T> leftFileReader,
+ ParquetReaderIterator<T> rightFileReader, Function<Pair<T,T>, T> mergeFunction) {
+ this.leftFileReader = leftFileReader;
+ this.rightFileReader = rightFileReader;
+ this.mergeFunction = mergeFunction;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return leftFileReader.hasNext();
Review comment:
Why is it ok to just base this on left
##########
File path: hudi-client/src/main/java/org/apache/hudi/config/HoodieBootstrapConfig.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.config;
+
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPathTranslator;
+import org.apache.hudi.common.config.DefaultHoodieConfig;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Bootstrap specific configs.
+ */
+public class HoodieBootstrapConfig extends DefaultHoodieConfig {
+
+ public static final String SOURCE_BASE_PATH_PROP = "hoodie.bootstrap.source.base.path";
+ public static final String BOOTSTRAP_MODE_SELECTOR = "hoodie.bootstrap.mode.selector";
+ public static final String FULL_BOOTRAP_INPUT_PROVIDER = "hoodie.bootstrap.full.input.provider";
Review comment:
this is. fine I think.. something to think about is : we should be able to add support for bootstrapping from another hudi table even, easily. Imagine a scenario where someone wants to change from bloom index to hbase index.. are the abstractions setup to support something. like that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #1678: [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#discussion_r441179570
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
Review comment:
Can we push `begin()` to the constructor ?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
Review comment:
Typo: bootstrap to be performed
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,532 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.HashSet;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.MetadataBootstrapKeyGenerator;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodiePartitionMetadata;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.view.HoodieTableFileSystemView;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.TableNotFoundException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, MetadataBootstrapKeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
+ ParquetMetadataConverter.NO_FILTER);
+ MessageType parquetSchema = readFooter.getFileMetaData().getSchema();
+ avroSchema = new AvroSchemaConverter().convert(parquetSchema);
+ Schema recordKeySchema = HoodieAvroUtils.generateProjectionSchema(avroSchema,
+ keyGenerator.getTopLevelRecordKeyFields());
+ LOG.info("Schema to be used for reading record Keys :" + recordKeySchema);
+ AvroReadSupport.setAvroReadSchema(table.getHadoopConf(), recordKeySchema);
+ AvroReadSupport.setRequestedProjection(table.getHadoopConf(), recordKeySchema);
+
+ BoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void> wrapper = null;
+ try (ParquetReader<IndexedRecord> reader =
+ AvroParquetReader.<IndexedRecord>builder(sourceFilePath).withConf(table.getHadoopConf()).build()) {
+ wrapper = new SparkBoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void>(config,
+ new ParquetReaderIterator(reader), new BootstrapRecordWriter(bootstrapHandle), inp -> {
+ String recKey = keyGenerator.getRecordKey(inp);
+ GenericRecord gr = new GenericData.Record(HoodieAvroUtils.RECORD_KEY_SCHEMA);
+ gr.put(HoodieRecord.RECORD_KEY_METADATA_FIELD, recKey);
+ BootstrapRecordPayload payload = new BootstrapRecordPayload(gr);
+ HoodieRecord rec = new HoodieRecord(new HoodieKey(recKey, partitionPath), payload);
+ return rec;
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ bootstrapHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ BootstrapWriteStatus writeStatus = (BootstrapWriteStatus)bootstrapHandle.getWriteStatus();
+ BootstrapSourceFileMapping bootstrapSourceFileMapping = new BootstrapSourceFileMapping(
+ config.getBootstrapSourceBasePath(), srcPartitionPath, partitionPath,
+ srcFileStatus, writeStatus.getFileId());
+ writeStatus.setBootstrapSourceFileMapping(bootstrapSourceFileMapping);
+ return writeStatus;
+ }
+
+ /**
+ * Return Bootstrap Mode selections for partitions listed and figure out bootstrap Schema.
+ * @param metaClient Hoodie Table Meta Client.
+ * @return
+ * @throws IOException
+ */
+ private Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> listAndProcessSourcePartitions(
+ HoodieTableMetaClient metaClient) throws IOException {
+ //TODO: Added HoodieFilter for manually testing bootstrap from source hudi table. Needs to be reverted.
+ final PathFilter hoodieFilter = new HoodieROTablePathFilter();
+ List<Pair<String, List<HoodieFileStatus>>> folders =
Review comment:
Why would we not do this in this pr itself?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, KeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
+ ParquetMetadataConverter.NO_FILTER);
+ MessageType parquetSchema = readFooter.getFileMetaData().getSchema();
+ avroSchema = new AvroSchemaConverter().convert(parquetSchema);
+ Schema recordKeySchema = HoodieAvroUtils.generateProjectionSchema(avroSchema,
+ keyGenerator.getTopLevelRecordKeyFields());
+ LOG.info("Schema to be used for reading record Keys :" + recordKeySchema);
+ AvroReadSupport.setAvroReadSchema(table.getHadoopConf(), recordKeySchema);
+ AvroReadSupport.setRequestedProjection(table.getHadoopConf(), recordKeySchema);
+
+ BoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void> wrapper = null;
+ try (ParquetReader<IndexedRecord> reader =
+ AvroParquetReader.<IndexedRecord>builder(sourceFilePath).withConf(table.getHadoopConf()).build()) {
+ wrapper = new SparkBoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void>(config,
Review comment:
can we avoid all this for the bootstrap path? do we really know if the InMemoryExecutor etc actually helps ?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/MergeHelper.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.commit;
+
+import java.io.ByteArrayOutputStream;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.utils.MergingParquetIterator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieMergeHandle;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.schema.MessageType;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Helper to read records from previous version of parquet and run Merge.
+ */
+public class MergeHelper {
+
+ /**
+ * Read records from previous version of base file and merge.
+ * @param table Hoodie Table
+ * @param upsertHandle Merge Handle
+ * @param <T>
+ * @throws IOException in case of error
+ */
+ public static <T extends HoodieRecordPayload<T>> void runMerge(HoodieTable<T> table,
+ HoodieMergeHandle<T> upsertHandle) throws IOException {
+ final boolean externalchemaTransformation = table.getConfig().shouldUseExternalSchemaTransformation();
+ Configuration configForHudiFile = new Configuration(table.getHadoopConf());
Review comment:
lets stick to `Hoodie` in code.
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/MergeHelper.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.commit;
+
+import java.io.ByteArrayOutputStream;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.utils.MergingParquetIterator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieMergeHandle;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.schema.MessageType;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Helper to read records from previous version of parquet and run Merge.
+ */
+public class MergeHelper {
+
+ /**
+ * Read records from previous version of base file and merge.
+ * @param table Hoodie Table
+ * @param upsertHandle Merge Handle
+ * @param <T>
+ * @throws IOException in case of error
+ */
+ public static <T extends HoodieRecordPayload<T>> void runMerge(HoodieTable<T> table,
+ HoodieMergeHandle<T> upsertHandle) throws IOException {
+ final boolean externalchemaTransformation = table.getConfig().shouldUseExternalSchemaTransformation();
+ Configuration configForHudiFile = new Configuration(table.getHadoopConf());
+ HoodieBaseFile baseFile = upsertHandle.getPrevBaseFile();
+
+ final GenericDatumWriter<GenericRecord> gWriter;
+ final GenericDatumReader<GenericRecord> gReader;
+ if (externalchemaTransformation || baseFile.getExternalBaseFile().isPresent()) {
+ MessageType usedParquetSchema = ParquetUtils.readSchema(table.getHadoopConf(), upsertHandle.getOldFilePath());
+ Schema lastWrittenSchema = new AvroSchemaConverter().convert(usedParquetSchema);
+ AvroReadSupport.setAvroReadSchema(configForHudiFile, lastWrittenSchema);
+ gWriter = new GenericDatumWriter<>(lastWrittenSchema);
+ gReader = new GenericDatumReader<>(lastWrittenSchema, upsertHandle.getWriterSchema());
+ } else {
+ gReader = null;
+ gWriter = null;
+ AvroReadSupport.setAvroReadSchema(configForHudiFile, upsertHandle.getWriterSchema());
+ }
+
+ BoundedInMemoryExecutor<GenericRecord, GenericRecord, Void> wrapper = null;
+ ParquetReader<GenericRecord> reader =
+ AvroParquetReader.<GenericRecord>builder(upsertHandle.getOldFilePath()).withConf(configForHudiFile).build();
+ ParquetReader<GenericRecord> externalFileReader = null;
+ try {
+ final Iterator<GenericRecord> readerIterator;
+ if (baseFile.getExternalBaseFile().isPresent()) {
+ readerIterator = getStitchedParquetIterator(table, upsertHandle, baseFile, reader,
+ externalchemaTransformation);
+ } else {
+ readerIterator = new ParquetReaderIterator(reader);
+ }
+
+ ThreadLocal<BinaryEncoder> encoderCache = new ThreadLocal<>();
+ ThreadLocal<BinaryDecoder> decoderCache = new ThreadLocal<>();
+ wrapper = new SparkBoundedInMemoryExecutor(table.getConfig(), readerIterator,
+ new UpdateHandler(upsertHandle), record -> {
+ if (!externalchemaTransformation) {
+ return record;
+ }
+ return transformRecordBasedOnNewSchema(gReader, gWriter, encoderCache, decoderCache,
+ (GenericRecord)record);
+
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ if (reader != null) {
+ reader.close();
+ }
+ upsertHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ }
+
+ private static GenericRecord transformRecordBasedOnNewSchema(
+ GenericDatumReader<GenericRecord> gReader, GenericDatumWriter<GenericRecord> gWriter,
+ ThreadLocal<BinaryEncoder> encoderCache, ThreadLocal<BinaryDecoder> decoderCache,
+ GenericRecord gRec) {
+ ByteArrayOutputStream inStream = null;
+ try {
+ inStream = new ByteArrayOutputStream();
+ BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(inStream, encoderCache.get());
+ encoderCache.set(encoder);
+ gWriter.write(gRec, encoder);
+ encoder.flush();
+ BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(inStream.toByteArray(), decoderCache.get());
Review comment:
lets use a line to separate logical blocks of code.. like a blank line between L134 and L135 will this much easier on the eyes?
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -326,4 +344,105 @@ public static String decompress(byte[] bytes) {
throw new HoodieIOException("IOException while decompressing text", e);
}
}
+
+ /**
+ * Generate a reader schema off the provided writeSchema, to just project out the provided columns.
+ */
+ public static Schema generateProjectionSchema(Schema originalSchema, List<String> fieldNames) {
Review comment:
lets make sure all of these have unit tests..
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/BootstrapIndex.java
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.ReflectionUtils;
+
+/**
+ * Bootstrap Index Interface.
Review comment:
base class.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/BootstrapIndex.java
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.ReflectionUtils;
+
+/**
+ * Bootstrap Index Interface.
+ */
+public abstract class BootstrapIndex implements Serializable {
+
+ protected static final long serialVersionUID = 1L;
+
+ protected final HoodieTableMetaClient metaClient;
+
+ public BootstrapIndex(HoodieTableMetaClient metaClient) {
+ this.metaClient = metaClient;
+ }
+
+ /**
+ * Create Bootstrap Index Reader.
+ * @return Index Reader
+ */
+ public abstract IndexReader createReader();
Review comment:
lets always use the BootstrapPrefix in the name.. else it will get confusing with real index
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/BootstrapIndex.java
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.ReflectionUtils;
+
+/**
+ * Bootstrap Index Interface.
+ */
+public abstract class BootstrapIndex implements Serializable {
+
+ protected static final long serialVersionUID = 1L;
+
+ protected final HoodieTableMetaClient metaClient;
+
+ public BootstrapIndex(HoodieTableMetaClient metaClient) {
+ this.metaClient = metaClient;
+ }
+
+ /**
+ * Create Bootstrap Index Reader.
+ * @return Index Reader
+ */
+ public abstract IndexReader createReader();
+
+ /**
+ * Create Bootstrap Index Writer.
+ * @param sourceBasePath Source Base Path
+ * @return Index Writer
+ */
+ public abstract IndexWriter createWriter(String sourceBasePath);
+
+ /**
+ * Drop bootstrap index.
+ */
+ public abstract void dropIndex();
+
+ /**
+ * Returns true if valid metadata bootstrap is present.
+ * @return
+ */
+ public final boolean isIndexAvailable() {
Review comment:
checkIndexAvailable()... since it actually does the checking .
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, KeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
Review comment:
Let’s not access parquet directly at this layer. We need a ReadHandle to delegate this to.. this layer should not be aware of parquet?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
Review comment:
replace `w->w` with Function.identity or something?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, KeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
+ ParquetMetadataConverter.NO_FILTER);
+ MessageType parquetSchema = readFooter.getFileMetaData().getSchema();
+ avroSchema = new AvroSchemaConverter().convert(parquetSchema);
+ Schema recordKeySchema = HoodieAvroUtils.generateProjectionSchema(avroSchema,
+ keyGenerator.getTopLevelRecordKeyFields());
+ LOG.info("Schema to be used for reading record Keys :" + recordKeySchema);
+ AvroReadSupport.setAvroReadSchema(table.getHadoopConf(), recordKeySchema);
+ AvroReadSupport.setRequestedProjection(table.getHadoopConf(), recordKeySchema);
+
+ BoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void> wrapper = null;
+ try (ParquetReader<IndexedRecord> reader =
+ AvroParquetReader.<IndexedRecord>builder(sourceFilePath).withConf(table.getHadoopConf()).build()) {
+ wrapper = new SparkBoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void>(config,
+ new ParquetReaderIterator(reader), new BootstrapRecordWriter(bootstrapHandle), inp -> {
+ String recKey = keyGenerator.getRecordKey(inp);
+ GenericRecord gr = new GenericData.Record(HoodieAvroUtils.RECORD_KEY_SCHEMA);
+ gr.put(HoodieRecord.RECORD_KEY_METADATA_FIELD, recKey);
+ BootstrapRecordPayload payload = new BootstrapRecordPayload(gr);
+ HoodieRecord rec = new HoodieRecord(new HoodieKey(recKey, partitionPath), payload);
+ return rec;
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ bootstrapHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ BootstrapWriteStatus writeStatus = (BootstrapWriteStatus)bootstrapHandle.getWriteStatus();
+ BootstrapSourceFileMapping bootstrapSourceFileMapping = new BootstrapSourceFileMapping(
+ config.getBootstrapSourceBasePath(), srcPartitionPath, partitionPath,
+ srcFileStatus, writeStatus.getFileId());
+ writeStatus.setBootstrapSourceFileMapping(bootstrapSourceFileMapping);
+ return writeStatus;
+ }
+
+ /**
+ * Return Bootstrap Mode selections for partitions listed and figure out bootstrap Schema.
+ * @param metaClient Hoodie Table Meta Client.
+ * @return
+ * @throws IOException
+ */
+ private Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> listAndProcessSourcePartitions(
+ HoodieTableMetaClient metaClient) throws IOException {
+ FileSystem fs = new Path(config.getBootstrapSourceBasePath()).getFileSystem(jsc.hadoopConfiguration());
+ List<Pair<String, List<HoodieFileStatus>>> folders =
+ FSUtils.getAllLeafFoldersWithFiles(bootstrapSourceFileSystem,
Review comment:
Like I mentioned in the other comments, we need to have an abstraction here to make the partition and file selection luggable
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, KeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
+ ParquetMetadataConverter.NO_FILTER);
+ MessageType parquetSchema = readFooter.getFileMetaData().getSchema();
+ avroSchema = new AvroSchemaConverter().convert(parquetSchema);
+ Schema recordKeySchema = HoodieAvroUtils.generateProjectionSchema(avroSchema,
+ keyGenerator.getTopLevelRecordKeyFields());
+ LOG.info("Schema to be used for reading record Keys :" + recordKeySchema);
+ AvroReadSupport.setAvroReadSchema(table.getHadoopConf(), recordKeySchema);
+ AvroReadSupport.setRequestedProjection(table.getHadoopConf(), recordKeySchema);
+
+ BoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void> wrapper = null;
+ try (ParquetReader<IndexedRecord> reader =
+ AvroParquetReader.<IndexedRecord>builder(sourceFilePath).withConf(table.getHadoopConf()).build()) {
+ wrapper = new SparkBoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void>(config,
+ new ParquetReaderIterator(reader), new BootstrapRecordWriter(bootstrapHandle), inp -> {
+ String recKey = keyGenerator.getRecordKey(inp);
+ GenericRecord gr = new GenericData.Record(HoodieAvroUtils.RECORD_KEY_SCHEMA);
+ gr.put(HoodieRecord.RECORD_KEY_METADATA_FIELD, recKey);
+ BootstrapRecordPayload payload = new BootstrapRecordPayload(gr);
+ HoodieRecord rec = new HoodieRecord(new HoodieKey(recKey, partitionPath), payload);
+ return rec;
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ bootstrapHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ BootstrapWriteStatus writeStatus = (BootstrapWriteStatus)bootstrapHandle.getWriteStatus();
+ BootstrapSourceFileMapping bootstrapSourceFileMapping = new BootstrapSourceFileMapping(
+ config.getBootstrapSourceBasePath(), srcPartitionPath, partitionPath,
+ srcFileStatus, writeStatus.getFileId());
+ writeStatus.setBootstrapSourceFileMapping(bootstrapSourceFileMapping);
+ return writeStatus;
+ }
+
+ /**
+ * Return Bootstrap Mode selections for partitions listed and figure out bootstrap Schema.
+ * @param metaClient Hoodie Table Meta Client.
+ * @return
+ * @throws IOException
+ */
+ private Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> listAndProcessSourcePartitions(
+ HoodieTableMetaClient metaClient) throws IOException {
+ FileSystem fs = new Path(config.getBootstrapSourceBasePath()).getFileSystem(jsc.hadoopConfiguration());
+ List<Pair<String, List<HoodieFileStatus>>> folders =
+ FSUtils.getAllLeafFoldersWithFiles(bootstrapSourceFileSystem,
+ config.getBootstrapSourceBasePath(), new PathFilter() {
+ @Override
+ public boolean accept(Path path) {
+ // TODO: Needs to be abstracted out when supporting different formats
+ // TODO: Remove hoodieFilter
+ return path.getName().endsWith(".parquet");
+ }
+ });
+
+ LOG.info("Fetching Bootstrap Schema !!");
+ BootstrapSourceSchemaProvider sourceSchemaProvider = new BootstrapSourceSchemaProvider(config);
+ bootstrapSchema = sourceSchemaProvider.getBootstrapSchema(jsc, folders).toString();
+ LOG.info("Bootstrap Schema :" + bootstrapSchema);
+
+ BootstrapModeSelector selector =
+ (BootstrapModeSelector) ReflectionUtils.loadClass(config.getBootstrapModeSelectorClass(), config);
+
+ Map<BootstrapMode, List<String>> result = selector.select(folders);
+ Map<String, List<HoodieFileStatus>> partitionToFiles = folders.stream().collect(
+ Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ // Ensure all partitions are accounted for
+ ValidationUtils.checkArgument(partitionToFiles.keySet().equals(
+ result.values().stream().flatMap(Collection::stream).collect(Collectors.toSet())));
+
+ return result.entrySet().stream().map(e -> Pair.of(e.getKey(), e.getValue().stream()
+ .map(p -> Pair.of(p, partitionToFiles.get(p))).collect(Collectors.toList())))
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+ }
+
+ private JavaRDD<BootstrapWriteStatus> runMetadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitions) {
+ if (null == partitions || partitions.isEmpty()) {
+ return jsc.emptyRDD();
+ }
+
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ KeyGenerator keyGenerator = (KeyGenerator) ReflectionUtils.loadClass(config.getBootstrapKeyGeneratorClass(),
+ properties);
+ MetadataBootstrapPartitionPathTranslator translator =
+ (MetadataBootstrapPartitionPathTranslator) ReflectionUtils.loadClass(
+ config.getBootstrapPartitionPathTranslatorClass(), properties);
+
+ return jsc.parallelize(partitions.stream()
+ .map(p -> Pair.of(p.getKey(), Pair.of(translator.getBootstrapTranslatedPath(p.getKey()), p.getValue())))
+ .flatMap(p -> p.getValue().getValue().stream()
+ .map(f -> Pair.of(p.getLeft(), Pair.of(p.getRight().getLeft(), f))))
+ .collect(Collectors.toList()),
+ config.getBootstrapParallelism())
+ .map(partitionFsPair -> {
+ return handleMetadataBootstrap(partitionFsPair.getLeft(), partitionFsPair.getRight().getLeft(),
+ partitionFsPair.getRight().getRight(), keyGenerator);
+ });
+ }
+
+ //TODO: Once we decouple commit protocol, we should change the class hierarchy to avoid doing this.
+ @Override
+ protected Partitioner getUpsertPartitioner(WorkloadProfile profile) {
+ return null;
Review comment:
do you expect this to be called? if not, lets throw `UnsupportedException`.. returning null should be a last resort anywhere. wdyt
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
Review comment:
Really gratifying to see this :)
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
Review comment:
i see how all the refactoring has helped make this more compact now :)
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
Review comment:
rename this to `close()`, consistent with things that are named writer?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
Review comment:
lets lose the `Done` ?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapRecordWriter.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+
+import java.io.IOException;
+
+/**
+ * Consumer that dequeues records from queue and sends to Merge Handle for writing.
+ */
+public class BootstrapRecordWriter extends BoundedInMemoryQueueConsumer<HoodieRecord, Void> {
+
+ private final HoodieBootstrapHandle bootstrapHandle;
+
+ public BootstrapRecordWriter(HoodieBootstrapHandle bootstrapHandle) {
+ this.bootstrapHandle = bootstrapHandle;
+ }
+
+ @Override
+ protected void consumeOneRecord(HoodieRecord record) {
+ try {
+ bootstrapHandle.write(record, record.getData().getInsertValue(bootstrapHandle.getWriterSchema()));
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ }
+
+ @Override
+ protected void finish() {}
+
+ @Override
+ protected Void getResult() {
+ return null;
Review comment:
or throw exception?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
Review comment:
Are these two different commits? Guess that makes it easier for us to special case
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
Review comment:
We should clarify somewhere that bootstrap file scheme and the real base path cannot be different. We can bootstrap fine but while querying don’t we need them to be the same hdfs, hdfs or s3,s3
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, KeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
+ ParquetMetadataConverter.NO_FILTER);
+ MessageType parquetSchema = readFooter.getFileMetaData().getSchema();
+ avroSchema = new AvroSchemaConverter().convert(parquetSchema);
+ Schema recordKeySchema = HoodieAvroUtils.generateProjectionSchema(avroSchema,
+ keyGenerator.getTopLevelRecordKeyFields());
+ LOG.info("Schema to be used for reading record Keys :" + recordKeySchema);
+ AvroReadSupport.setAvroReadSchema(table.getHadoopConf(), recordKeySchema);
+ AvroReadSupport.setRequestedProjection(table.getHadoopConf(), recordKeySchema);
+
+ BoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void> wrapper = null;
+ try (ParquetReader<IndexedRecord> reader =
+ AvroParquetReader.<IndexedRecord>builder(sourceFilePath).withConf(table.getHadoopConf()).build()) {
+ wrapper = new SparkBoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void>(config,
+ new ParquetReaderIterator(reader), new BootstrapRecordWriter(bootstrapHandle), inp -> {
+ String recKey = keyGenerator.getRecordKey(inp);
+ GenericRecord gr = new GenericData.Record(HoodieAvroUtils.RECORD_KEY_SCHEMA);
+ gr.put(HoodieRecord.RECORD_KEY_METADATA_FIELD, recKey);
+ BootstrapRecordPayload payload = new BootstrapRecordPayload(gr);
+ HoodieRecord rec = new HoodieRecord(new HoodieKey(recKey, partitionPath), payload);
+ return rec;
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ bootstrapHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ BootstrapWriteStatus writeStatus = (BootstrapWriteStatus)bootstrapHandle.getWriteStatus();
+ BootstrapSourceFileMapping bootstrapSourceFileMapping = new BootstrapSourceFileMapping(
+ config.getBootstrapSourceBasePath(), srcPartitionPath, partitionPath,
+ srcFileStatus, writeStatus.getFileId());
+ writeStatus.setBootstrapSourceFileMapping(bootstrapSourceFileMapping);
+ return writeStatus;
+ }
+
+ /**
+ * Return Bootstrap Mode selections for partitions listed and figure out bootstrap Schema.
+ * @param metaClient Hoodie Table Meta Client.
+ * @return
+ * @throws IOException
+ */
+ private Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> listAndProcessSourcePartitions(
+ HoodieTableMetaClient metaClient) throws IOException {
+ FileSystem fs = new Path(config.getBootstrapSourceBasePath()).getFileSystem(jsc.hadoopConfiguration());
+ List<Pair<String, List<HoodieFileStatus>>> folders =
+ FSUtils.getAllLeafFoldersWithFiles(bootstrapSourceFileSystem,
+ config.getBootstrapSourceBasePath(), new PathFilter() {
+ @Override
+ public boolean accept(Path path) {
+ // TODO: Needs to be abstracted out when supporting different formats
+ // TODO: Remove hoodieFilter
+ return path.getName().endsWith(".parquet");
+ }
+ });
+
+ LOG.info("Fetching Bootstrap Schema !!");
+ BootstrapSourceSchemaProvider sourceSchemaProvider = new BootstrapSourceSchemaProvider(config);
+ bootstrapSchema = sourceSchemaProvider.getBootstrapSchema(jsc, folders).toString();
+ LOG.info("Bootstrap Schema :" + bootstrapSchema);
+
+ BootstrapModeSelector selector =
+ (BootstrapModeSelector) ReflectionUtils.loadClass(config.getBootstrapModeSelectorClass(), config);
+
+ Map<BootstrapMode, List<String>> result = selector.select(folders);
+ Map<String, List<HoodieFileStatus>> partitionToFiles = folders.stream().collect(
+ Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ // Ensure all partitions are accounted for
+ ValidationUtils.checkArgument(partitionToFiles.keySet().equals(
+ result.values().stream().flatMap(Collection::stream).collect(Collectors.toSet())));
+
+ return result.entrySet().stream().map(e -> Pair.of(e.getKey(), e.getValue().stream()
Review comment:
make this more readable? like descriptive name for `e`?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
Review comment:
Rename to validate()? There are no arguments per se here
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/MergeHelper.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.commit;
+
+import java.io.ByteArrayOutputStream;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.utils.MergingParquetIterator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieMergeHandle;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.schema.MessageType;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Helper to read records from previous version of parquet and run Merge.
+ */
+public class MergeHelper {
+
+ /**
+ * Read records from previous version of base file and merge.
+ * @param table Hoodie Table
+ * @param upsertHandle Merge Handle
+ * @param <T>
+ * @throws IOException in case of error
+ */
+ public static <T extends HoodieRecordPayload<T>> void runMerge(HoodieTable<T> table,
+ HoodieMergeHandle<T> upsertHandle) throws IOException {
+ final boolean externalchemaTransformation = table.getConfig().shouldUseExternalSchemaTransformation();
Review comment:
typo: externalSchema...
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
Review comment:
can we avoid using `null` as a sentinel.. we should ideally add a checkstyle for this.
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, KeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
Review comment:
I see you are reading the source data files.. still some checks for supported format types/modularization needed to support other file formats?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/MergeHelper.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.commit;
+
+import java.io.ByteArrayOutputStream;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.utils.MergingParquetIterator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieMergeHandle;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.schema.MessageType;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Helper to read records from previous version of parquet and run Merge.
+ */
+public class MergeHelper {
+
+ /**
+ * Read records from previous version of base file and merge.
+ * @param table Hoodie Table
+ * @param upsertHandle Merge Handle
+ * @param <T>
+ * @throws IOException in case of error
+ */
+ public static <T extends HoodieRecordPayload<T>> void runMerge(HoodieTable<T> table,
+ HoodieMergeHandle<T> upsertHandle) throws IOException {
+ final boolean externalchemaTransformation = table.getConfig().shouldUseExternalSchemaTransformation();
+ Configuration configForHudiFile = new Configuration(table.getHadoopConf());
+ HoodieBaseFile baseFile = upsertHandle.getPrevBaseFile();
+
+ final GenericDatumWriter<GenericRecord> gWriter;
+ final GenericDatumReader<GenericRecord> gReader;
+ if (externalchemaTransformation || baseFile.getExternalBaseFile().isPresent()) {
+ MessageType usedParquetSchema = ParquetUtils.readSchema(table.getHadoopConf(), upsertHandle.getOldFilePath());
+ Schema lastWrittenSchema = new AvroSchemaConverter().convert(usedParquetSchema);
+ AvroReadSupport.setAvroReadSchema(configForHudiFile, lastWrittenSchema);
+ gWriter = new GenericDatumWriter<>(lastWrittenSchema);
+ gReader = new GenericDatumReader<>(lastWrittenSchema, upsertHandle.getWriterSchema());
+ } else {
+ gReader = null;
+ gWriter = null;
+ AvroReadSupport.setAvroReadSchema(configForHudiFile, upsertHandle.getWriterSchema());
+ }
+
+ BoundedInMemoryExecutor<GenericRecord, GenericRecord, Void> wrapper = null;
+ ParquetReader<GenericRecord> reader =
+ AvroParquetReader.<GenericRecord>builder(upsertHandle.getOldFilePath()).withConf(configForHudiFile).build();
+ ParquetReader<GenericRecord> externalFileReader = null;
+ try {
+ final Iterator<GenericRecord> readerIterator;
+ if (baseFile.getExternalBaseFile().isPresent()) {
+ readerIterator = getStitchedParquetIterator(table, upsertHandle, baseFile, reader,
+ externalchemaTransformation);
+ } else {
+ readerIterator = new ParquetReaderIterator(reader);
+ }
+
+ ThreadLocal<BinaryEncoder> encoderCache = new ThreadLocal<>();
+ ThreadLocal<BinaryDecoder> decoderCache = new ThreadLocal<>();
+ wrapper = new SparkBoundedInMemoryExecutor(table.getConfig(), readerIterator,
+ new UpdateHandler(upsertHandle), record -> {
+ if (!externalchemaTransformation) {
+ return record;
+ }
+ return transformRecordBasedOnNewSchema(gReader, gWriter, encoderCache, decoderCache,
+ (GenericRecord)record);
+
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ if (reader != null) {
+ reader.close();
+ }
+ upsertHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ }
+
+ private static GenericRecord transformRecordBasedOnNewSchema(
+ GenericDatumReader<GenericRecord> gReader, GenericDatumWriter<GenericRecord> gWriter,
+ ThreadLocal<BinaryEncoder> encoderCache, ThreadLocal<BinaryDecoder> decoderCache,
+ GenericRecord gRec) {
+ ByteArrayOutputStream inStream = null;
+ try {
+ inStream = new ByteArrayOutputStream();
+ BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(inStream, encoderCache.get());
+ encoderCache.set(encoder);
+ gWriter.write(gRec, encoder);
+ encoder.flush();
+ BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(inStream.toByteArray(), decoderCache.get());
+ decoderCache.set(decoder);
+ GenericRecord transformedRec = gReader.read(null, decoder);
+ return transformedRec;
+ } catch (IOException e) {
+ throw new HoodieException(e);
+ } finally {
+ try {
+ inStream.close();
+ } catch (IOException ioe) {
+ throw new HoodieException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+
+
+ /**
+ * Create Parquet record iterator that provides a stitched view of record read from skeleton and external file.
+ * @param table Hoodie Table
+ * @param upsertHandle Merge Handle
+ * @param baseFile Base File
+ * @param reader Record Reader corresponding to skeleton file
+ * @param externalchemaTransformation Disable Parquet managed avro schema evolution
Review comment:
typo in variable name..
##########
File path: hudi-client/src/test/java/org/apache/hudi/client/TestTableSchemaEvolution.java
##########
@@ -139,7 +139,7 @@ public void testMORTable() throws Exception {
// Create the table
HoodieTableMetaClient.initTableType(metaClient.getHadoopConf(), metaClient.getBasePath(),
HoodieTableType.MERGE_ON_READ, metaClient.getTableConfig().getTableName(),
- metaClient.getArchivePath(), metaClient.getTableConfig().getPayloadClass(), VERSION_1);
+ metaClient.getArchivePath(), metaClient.getTableConfig().getPayloadClass(), VERSION_1, null);
Review comment:
passing null is also something we can avoid if needed..
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, KeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
+ ParquetMetadataConverter.NO_FILTER);
+ MessageType parquetSchema = readFooter.getFileMetaData().getSchema();
+ avroSchema = new AvroSchemaConverter().convert(parquetSchema);
+ Schema recordKeySchema = HoodieAvroUtils.generateProjectionSchema(avroSchema,
+ keyGenerator.getTopLevelRecordKeyFields());
+ LOG.info("Schema to be used for reading record Keys :" + recordKeySchema);
+ AvroReadSupport.setAvroReadSchema(table.getHadoopConf(), recordKeySchema);
+ AvroReadSupport.setRequestedProjection(table.getHadoopConf(), recordKeySchema);
+
+ BoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void> wrapper = null;
+ try (ParquetReader<IndexedRecord> reader =
+ AvroParquetReader.<IndexedRecord>builder(sourceFilePath).withConf(table.getHadoopConf()).build()) {
+ wrapper = new SparkBoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void>(config,
+ new ParquetReaderIterator(reader), new BootstrapRecordWriter(bootstrapHandle), inp -> {
+ String recKey = keyGenerator.getRecordKey(inp);
+ GenericRecord gr = new GenericData.Record(HoodieAvroUtils.RECORD_KEY_SCHEMA);
+ gr.put(HoodieRecord.RECORD_KEY_METADATA_FIELD, recKey);
+ BootstrapRecordPayload payload = new BootstrapRecordPayload(gr);
+ HoodieRecord rec = new HoodieRecord(new HoodieKey(recKey, partitionPath), payload);
+ return rec;
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ bootstrapHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ BootstrapWriteStatus writeStatus = (BootstrapWriteStatus)bootstrapHandle.getWriteStatus();
+ BootstrapSourceFileMapping bootstrapSourceFileMapping = new BootstrapSourceFileMapping(
+ config.getBootstrapSourceBasePath(), srcPartitionPath, partitionPath,
+ srcFileStatus, writeStatus.getFileId());
+ writeStatus.setBootstrapSourceFileMapping(bootstrapSourceFileMapping);
+ return writeStatus;
+ }
+
+ /**
+ * Return Bootstrap Mode selections for partitions listed and figure out bootstrap Schema.
+ * @param metaClient Hoodie Table Meta Client.
+ * @return
+ * @throws IOException
+ */
+ private Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> listAndProcessSourcePartitions(
+ HoodieTableMetaClient metaClient) throws IOException {
+ FileSystem fs = new Path(config.getBootstrapSourceBasePath()).getFileSystem(jsc.hadoopConfiguration());
+ List<Pair<String, List<HoodieFileStatus>>> folders =
+ FSUtils.getAllLeafFoldersWithFiles(bootstrapSourceFileSystem,
+ config.getBootstrapSourceBasePath(), new PathFilter() {
+ @Override
+ public boolean accept(Path path) {
+ // TODO: Needs to be abstracted out when supporting different formats
+ // TODO: Remove hoodieFilter
+ return path.getName().endsWith(".parquet");
+ }
+ });
+
+ LOG.info("Fetching Bootstrap Schema !!");
+ BootstrapSourceSchemaProvider sourceSchemaProvider = new BootstrapSourceSchemaProvider(config);
+ bootstrapSchema = sourceSchemaProvider.getBootstrapSchema(jsc, folders).toString();
+ LOG.info("Bootstrap Schema :" + bootstrapSchema);
+
+ BootstrapModeSelector selector =
+ (BootstrapModeSelector) ReflectionUtils.loadClass(config.getBootstrapModeSelectorClass(), config);
+
+ Map<BootstrapMode, List<String>> result = selector.select(folders);
+ Map<String, List<HoodieFileStatus>> partitionToFiles = folders.stream().collect(
+ Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ // Ensure all partitions are accounted for
+ ValidationUtils.checkArgument(partitionToFiles.keySet().equals(
+ result.values().stream().flatMap(Collection::stream).collect(Collectors.toSet())));
+
+ return result.entrySet().stream().map(e -> Pair.of(e.getKey(), e.getValue().stream()
+ .map(p -> Pair.of(p, partitionToFiles.get(p))).collect(Collectors.toList())))
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+ }
+
+ private JavaRDD<BootstrapWriteStatus> runMetadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitions) {
+ if (null == partitions || partitions.isEmpty()) {
+ return jsc.emptyRDD();
+ }
+
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ KeyGenerator keyGenerator = (KeyGenerator) ReflectionUtils.loadClass(config.getBootstrapKeyGeneratorClass(),
+ properties);
+ MetadataBootstrapPartitionPathTranslator translator =
+ (MetadataBootstrapPartitionPathTranslator) ReflectionUtils.loadClass(
+ config.getBootstrapPartitionPathTranslatorClass(), properties);
+
+ return jsc.parallelize(partitions.stream()
Review comment:
this piece of code is very hard to read.. Can we please simplify this?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, KeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
+ ParquetMetadataConverter.NO_FILTER);
+ MessageType parquetSchema = readFooter.getFileMetaData().getSchema();
+ avroSchema = new AvroSchemaConverter().convert(parquetSchema);
+ Schema recordKeySchema = HoodieAvroUtils.generateProjectionSchema(avroSchema,
+ keyGenerator.getTopLevelRecordKeyFields());
+ LOG.info("Schema to be used for reading record Keys :" + recordKeySchema);
+ AvroReadSupport.setAvroReadSchema(table.getHadoopConf(), recordKeySchema);
+ AvroReadSupport.setRequestedProjection(table.getHadoopConf(), recordKeySchema);
+
+ BoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void> wrapper = null;
+ try (ParquetReader<IndexedRecord> reader =
+ AvroParquetReader.<IndexedRecord>builder(sourceFilePath).withConf(table.getHadoopConf()).build()) {
+ wrapper = new SparkBoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void>(config,
+ new ParquetReaderIterator(reader), new BootstrapRecordWriter(bootstrapHandle), inp -> {
+ String recKey = keyGenerator.getRecordKey(inp);
+ GenericRecord gr = new GenericData.Record(HoodieAvroUtils.RECORD_KEY_SCHEMA);
+ gr.put(HoodieRecord.RECORD_KEY_METADATA_FIELD, recKey);
+ BootstrapRecordPayload payload = new BootstrapRecordPayload(gr);
+ HoodieRecord rec = new HoodieRecord(new HoodieKey(recKey, partitionPath), payload);
+ return rec;
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ bootstrapHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ BootstrapWriteStatus writeStatus = (BootstrapWriteStatus)bootstrapHandle.getWriteStatus();
+ BootstrapSourceFileMapping bootstrapSourceFileMapping = new BootstrapSourceFileMapping(
+ config.getBootstrapSourceBasePath(), srcPartitionPath, partitionPath,
+ srcFileStatus, writeStatus.getFileId());
+ writeStatus.setBootstrapSourceFileMapping(bootstrapSourceFileMapping);
+ return writeStatus;
+ }
+
+ /**
+ * Return Bootstrap Mode selections for partitions listed and figure out bootstrap Schema.
+ * @param metaClient Hoodie Table Meta Client.
+ * @return
+ * @throws IOException
+ */
+ private Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> listAndProcessSourcePartitions(
+ HoodieTableMetaClient metaClient) throws IOException {
+ FileSystem fs = new Path(config.getBootstrapSourceBasePath()).getFileSystem(jsc.hadoopConfiguration());
+ List<Pair<String, List<HoodieFileStatus>>> folders =
+ FSUtils.getAllLeafFoldersWithFiles(bootstrapSourceFileSystem,
+ config.getBootstrapSourceBasePath(), new PathFilter() {
+ @Override
+ public boolean accept(Path path) {
+ // TODO: Needs to be abstracted out when supporting different formats
+ // TODO: Remove hoodieFilter
+ return path.getName().endsWith(".parquet");
+ }
+ });
+
+ LOG.info("Fetching Bootstrap Schema !!");
+ BootstrapSourceSchemaProvider sourceSchemaProvider = new BootstrapSourceSchemaProvider(config);
+ bootstrapSchema = sourceSchemaProvider.getBootstrapSchema(jsc, folders).toString();
+ LOG.info("Bootstrap Schema :" + bootstrapSchema);
+
+ BootstrapModeSelector selector =
+ (BootstrapModeSelector) ReflectionUtils.loadClass(config.getBootstrapModeSelectorClass(), config);
+
+ Map<BootstrapMode, List<String>> result = selector.select(folders);
+ Map<String, List<HoodieFileStatus>> partitionToFiles = folders.stream().collect(
+ Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ // Ensure all partitions are accounted for
+ ValidationUtils.checkArgument(partitionToFiles.keySet().equals(
+ result.values().stream().flatMap(Collection::stream).collect(Collectors.toSet())));
+
+ return result.entrySet().stream().map(e -> Pair.of(e.getKey(), e.getValue().stream()
+ .map(p -> Pair.of(p, partitionToFiles.get(p))).collect(Collectors.toList())))
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+ }
+
+ private JavaRDD<BootstrapWriteStatus> runMetadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitions) {
+ if (null == partitions || partitions.isEmpty()) {
+ return jsc.emptyRDD();
+ }
+
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ KeyGenerator keyGenerator = (KeyGenerator) ReflectionUtils.loadClass(config.getBootstrapKeyGeneratorClass(),
+ properties);
+ MetadataBootstrapPartitionPathTranslator translator =
+ (MetadataBootstrapPartitionPathTranslator) ReflectionUtils.loadClass(
+ config.getBootstrapPartitionPathTranslatorClass(), properties);
+
+ return jsc.parallelize(partitions.stream()
+ .map(p -> Pair.of(p.getKey(), Pair.of(translator.getBootstrapTranslatedPath(p.getKey()), p.getValue())))
+ .flatMap(p -> p.getValue().getValue().stream()
+ .map(f -> Pair.of(p.getLeft(), Pair.of(p.getRight().getLeft(), f))))
+ .collect(Collectors.toList()),
+ config.getBootstrapParallelism())
+ .map(partitionFsPair -> {
+ return handleMetadataBootstrap(partitionFsPair.getLeft(), partitionFsPair.getRight().getLeft(),
+ partitionFsPair.getRight().getRight(), keyGenerator);
+ });
+ }
+
+ //TODO: Once we decouple commit protocol, we should change the class hierarchy to avoid doing this.
Review comment:
File a JIRA for this?
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapRecordWriter.java
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+
+import java.io.IOException;
+
+/**
+ * Consumer that dequeues records from queue and sends to Merge Handle for writing.
+ */
+public class BootstrapRecordWriter extends BoundedInMemoryQueueConsumer<HoodieRecord, Void> {
Review comment:
rename: `BootstrapRecordConsumer`
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/MergeHelper.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.commit;
+
+import java.io.ByteArrayOutputStream;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.utils.MergingParquetIterator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieMergeHandle;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.schema.MessageType;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Helper to read records from previous version of parquet and run Merge.
+ */
+public class MergeHelper {
Review comment:
I believe most of this class is code moved from the table with support for stitching? let me know if there are key pieces that need to be reviewed here.
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackActionExecutor.java
##########
@@ -189,4 +190,13 @@ protected void deleteInflightAndRequestedInstant(boolean deleteInstant,
LOG.warn("Rollback finished without deleting inflight instant file. Instant=" + instantToBeDeleted);
}
}
+
+ protected void dropBootstrapIndexIfNeeded(HoodieInstant instantToRollback) {
+ if (HoodieTimeline.compareTimestamps(instantToRollback.getTimestamp(), HoodieTimeline.EQUALS,
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS)) {
+ LOG.info("Dropping bootstrap index as metadata bootstrap commit is getting rolled back !!");
+ BootstrapIndex.getBootstrapIndex(table.getMetaClient()).dropIndex();
Review comment:
we need to ensure this is idempotent.. i.e can fail in the middle and still succeed when rolled back again..
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -64,10 +69,12 @@
private static ThreadLocal<BinaryDecoder> reuseDecoder = ThreadLocal.withInitial(() -> null);
// All metadata fields are optional strings.
- static final Schema METADATA_FIELD_SCHEMA =
+ public static final Schema METADATA_FIELD_SCHEMA =
Schema.createUnion(Arrays.asList(Schema.create(Schema.Type.NULL), Schema.create(Schema.Type.STRING)));
- private static final Schema RECORD_KEY_SCHEMA = initRecordKeySchema();
+ public static final Schema RECORD_KEY_SCHEMA = initRecordKeySchema();
+
+ public static final Integer NUM_HUDI_METADATA_COLS = 5;
Review comment:
should this belong with the actual metadata column names? may be high time we pulled those into a `enum` ? (ordering/size done in a standard way)
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/MergeHelper.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.commit;
+
+import java.io.ByteArrayOutputStream;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.utils.MergingParquetIterator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieMergeHandle;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.schema.MessageType;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Helper to read records from previous version of parquet and run Merge.
+ */
+public class MergeHelper {
+
+ /**
+ * Read records from previous version of base file and merge.
+ * @param table Hoodie Table
+ * @param upsertHandle Merge Handle
+ * @param <T>
+ * @throws IOException in case of error
+ */
+ public static <T extends HoodieRecordPayload<T>> void runMerge(HoodieTable<T> table,
+ HoodieMergeHandle<T> upsertHandle) throws IOException {
+ final boolean externalchemaTransformation = table.getConfig().shouldUseExternalSchemaTransformation();
+ Configuration configForHudiFile = new Configuration(table.getHadoopConf());
+ HoodieBaseFile baseFile = upsertHandle.getPrevBaseFile();
+
+ final GenericDatumWriter<GenericRecord> gWriter;
+ final GenericDatumReader<GenericRecord> gReader;
+ if (externalchemaTransformation || baseFile.getExternalBaseFile().isPresent()) {
+ MessageType usedParquetSchema = ParquetUtils.readSchema(table.getHadoopConf(), upsertHandle.getOldFilePath());
+ Schema lastWrittenSchema = new AvroSchemaConverter().convert(usedParquetSchema);
+ AvroReadSupport.setAvroReadSchema(configForHudiFile, lastWrittenSchema);
+ gWriter = new GenericDatumWriter<>(lastWrittenSchema);
+ gReader = new GenericDatumReader<>(lastWrittenSchema, upsertHandle.getWriterSchema());
+ } else {
+ gReader = null;
+ gWriter = null;
+ AvroReadSupport.setAvroReadSchema(configForHudiFile, upsertHandle.getWriterSchema());
+ }
+
+ BoundedInMemoryExecutor<GenericRecord, GenericRecord, Void> wrapper = null;
+ ParquetReader<GenericRecord> reader =
+ AvroParquetReader.<GenericRecord>builder(upsertHandle.getOldFilePath()).withConf(configForHudiFile).build();
+ ParquetReader<GenericRecord> externalFileReader = null;
+ try {
+ final Iterator<GenericRecord> readerIterator;
+ if (baseFile.getExternalBaseFile().isPresent()) {
+ readerIterator = getStitchedParquetIterator(table, upsertHandle, baseFile, reader,
+ externalchemaTransformation);
+ } else {
+ readerIterator = new ParquetReaderIterator(reader);
+ }
+
+ ThreadLocal<BinaryEncoder> encoderCache = new ThreadLocal<>();
+ ThreadLocal<BinaryDecoder> decoderCache = new ThreadLocal<>();
+ wrapper = new SparkBoundedInMemoryExecutor(table.getConfig(), readerIterator,
+ new UpdateHandler(upsertHandle), record -> {
+ if (!externalchemaTransformation) {
+ return record;
+ }
+ return transformRecordBasedOnNewSchema(gReader, gWriter, encoderCache, decoderCache,
+ (GenericRecord)record);
+
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ if (reader != null) {
+ reader.close();
+ }
+ upsertHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ }
+
+ private static GenericRecord transformRecordBasedOnNewSchema(
+ GenericDatumReader<GenericRecord> gReader, GenericDatumWriter<GenericRecord> gWriter,
+ ThreadLocal<BinaryEncoder> encoderCache, ThreadLocal<BinaryDecoder> decoderCache,
+ GenericRecord gRec) {
+ ByteArrayOutputStream inStream = null;
+ try {
+ inStream = new ByteArrayOutputStream();
+ BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(inStream, encoderCache.get());
+ encoderCache.set(encoder);
+ gWriter.write(gRec, encoder);
+ encoder.flush();
+ BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(inStream.toByteArray(), decoderCache.get());
+ decoderCache.set(decoder);
+ GenericRecord transformedRec = gReader.read(null, decoder);
+ return transformedRec;
+ } catch (IOException e) {
+ throw new HoodieException(e);
+ } finally {
+ try {
+ inStream.close();
+ } catch (IOException ioe) {
+ throw new HoodieException(ioe.getMessage(), ioe);
+ }
+ }
+ }
+
+
+ /**
+ * Create Parquet record iterator that provides a stitched view of record read from skeleton and external file.
+ * @param table Hoodie Table
+ * @param upsertHandle Merge Handle
+ * @param baseFile Base File
+ * @param reader Record Reader corresponding to skeleton file
+ * @param externalchemaTransformation Disable Parquet managed avro schema evolution
+ * @param <T>
+ * @return
+ * @throws IOException in case of error reading files
+ */
+ private static <T extends HoodieRecordPayload<T>> Iterator<GenericRecord> getStitchedParquetIterator(
+ HoodieTable<T> table, HoodieMergeHandle<T> upsertHandle, HoodieBaseFile baseFile,
+ ParquetReader<GenericRecord> reader, boolean externalchemaTransformation) throws IOException {
+ Path externalFilePath = new Path(baseFile.getExternalBaseFile().get().getPath());
+ Configuration configForExternalFile = new Configuration(table.getHadoopConf());
+ if (externalchemaTransformation) {
+ MessageType usedExtParquetSchema = ParquetUtils.readSchema(configForExternalFile, externalFilePath);
+ Schema lastExtWrittenSchema = new AvroSchemaConverter().convert(usedExtParquetSchema);
+ AvroReadSupport.setAvroReadSchema(configForExternalFile, lastExtWrittenSchema);
+ } else {
+ AvroReadSupport.setAvroReadSchema(configForExternalFile, upsertHandle.getOriginalSchema());
+ }
+ ParquetReader<GenericRecord> externalFileReader = AvroParquetReader.<GenericRecord>builder(externalFilePath).withConf(configForExternalFile).build();
Review comment:
Once again some abstraction needed to plug in other formats easily
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/commit/MergeHelper.java
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.commit;
+
+import java.io.ByteArrayOutputStream;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.avro.io.BinaryDecoder;
+import org.apache.avro.io.BinaryEncoder;
+import org.apache.avro.io.DecoderFactory;
+import org.apache.avro.io.EncoderFactory;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.client.utils.MergingParquetIterator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.model.HoodieBaseFile;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieMergeHandle;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.schema.MessageType;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.util.Iterator;
+
+/**
+ * Helper to read records from previous version of parquet and run Merge.
+ */
+public class MergeHelper {
Review comment:
we need an unit test for this class?
##########
File path: hudi-common/src/main/avro/HoodieCompactionOperation.avsc
##########
@@ -61,6 +61,11 @@
"values":"double"
}],
"default": null
+ },
+ {
+ "name":"externalDataFilePath",
Review comment:
why this is needed? again, lets either use "bootstrap" or "external"... one term consistently throughout.. this does cause a lot of debt in reading code in the longer term..
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
+ Option<Map<String, String>> extraMetadata) {
+ super(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withAutoCommit(true).withWriteStatusClass(BootstrapWriteStatus.class)
+ .withBulkInsertParallelism(config.getBootstrapParallelism())
+ .build(), table, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS, WriteOperationType.BOOTSTRAP,
+ extraMetadata);
+ bootstrapSourceFileSystem = FSUtils.getFs(config.getBootstrapSourceBasePath(), hadoopConf);
+ }
+
+ private void checkArguments() {
+ ValidationUtils.checkArgument(config.getBootstrapSourceBasePath() != null,
+ "Ensure Bootstrap Source Path is set");
+ ValidationUtils.checkArgument(config.getBootstrapModeSelectorClass() != null,
+ "Ensure Bootstrap Partition Selector is set");
+ ValidationUtils.checkArgument(config.getBootstrapKeyGeneratorClass() != null,
+ "Ensure bootstrap key generator class is set");
+ }
+
+ @Override
+ public HoodieBootstrapWriteMetadata execute() {
+ checkArguments();
+ try {
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ Option<HoodieInstant> completetedInstant =
+ metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant();
+ ValidationUtils.checkArgument(!completetedInstant.isPresent(),
+ "Active Timeline is expected to be empty for bootstrapped to be performed. "
+ + "If you want to re-bootstrap, please rollback bootstrap first !!");
+ Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> partitionSelections =
+ listAndProcessSourcePartitions(metaClient);
+
+ // First run metadata bootstrap which will implicitly commit
+ HoodieWriteMetadata metadataResult = metadataBootstrap(partitionSelections.get(
+ BootstrapMode.METADATA_ONLY_BOOTSTRAP));
+ // if there are full bootstrap to be performed, perform that too
+ HoodieWriteMetadata fullBootstrapResult =
+ fullBootstrap(partitionSelections.get(BootstrapMode.FULL_BOOTSTRAP));
+ return new HoodieBootstrapWriteMetadata(metadataResult, fullBootstrapResult);
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ }
+
+ protected String getSchemaToStoreInCommit() {
+ return bootstrapSchema;
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata metadataBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ metaClient.getActiveTimeline().createNewInstant(
+ new HoodieInstant(State.REQUESTED, metaClient.getCommitActionType(),
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS));
+
+ table.getActiveTimeline().transitionRequestedToInflight(new HoodieInstant(State.REQUESTED,
+ metaClient.getCommitActionType(), HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS), Option.empty());
+
+ JavaRDD<BootstrapWriteStatus> bootstrapWriteStatuses = runMetadataBootstrap(partitionFilesList);
+
+ HoodieWriteMetadata result = new HoodieWriteMetadata();
+ updateIndexAndCommitIfNeeded(bootstrapWriteStatuses.map(w -> w), result);
+ return result;
+ }
+
+ @Override
+ protected void commit(Option<Map<String, String>> extraMetadata, HoodieWriteMetadata result) {
+ // Perform bootstrap index write and then commit. Make sure both record-key and bootstrap-index
+ // is all done in a single job DAG.
+ Map<String, List<Pair<BootstrapSourceFileMapping, HoodieWriteStat>>> bootstrapSourceAndStats =
+ result.getWriteStatuses().collect().stream()
+ .map(w -> {
+ BootstrapWriteStatus ws = (BootstrapWriteStatus) w;
+ return Pair.of(ws.getBootstrapSourceFileMapping(), ws.getStat());
+ }).collect(Collectors.groupingBy(w -> w.getKey().getHudiPartitionPath()));
+ HoodieTableMetaClient metaClient = table.getMetaClient();
+ try (BootstrapIndex.IndexWriter indexWriter = BootstrapIndex.getBootstrapIndex(metaClient)
+ .createWriter(config.getBootstrapSourceBasePath())) {
+ LOG.info("Starting to write bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ indexWriter.begin();
+ bootstrapSourceAndStats.forEach((key, value) -> indexWriter.appendNextPartition(key,
+ value.stream().map(w -> w.getKey()).collect(Collectors.toList())));
+ indexWriter.finish();
+ LOG.info("Finished writing bootstrap index for source " + config.getBootstrapSourceBasePath() + " in table "
+ + config.getBasePath());
+ }
+
+ super.commit(extraMetadata, result, bootstrapSourceAndStats.values().stream()
+ .flatMap(f -> f.stream().map(Pair::getValue)).collect(Collectors.toList()));
+ LOG.info("Done Committing metadata bootstrap !!");
+ }
+
+ /**
+ * Perform Metadata Bootstrap.
+ * @param partitionFilesList List of partitions and files within that partitions
+ */
+ protected HoodieWriteMetadata fullBootstrap(List<Pair<String, List<HoodieFileStatus>>> partitionFilesList) {
+ if (null == partitionFilesList || partitionFilesList.isEmpty()) {
+ return null;
+ }
+ TypedProperties properties = new TypedProperties();
+ properties.putAll(config.getProps());
+ FullBootstrapInputProvider inputProvider =
+ (FullBootstrapInputProvider) ReflectionUtils.loadClass(config.getFullBootstrapInputProvider(),
+ properties, jsc);
+ JavaRDD<HoodieRecord> inputRecordsRDD =
+ inputProvider.generateInputRecordRDD("bootstrap_source", config.getBootstrapSourceBasePath(),
+ partitionFilesList);
+ // Start Full Bootstrap
+ final HoodieInstant requested = new HoodieInstant(State.REQUESTED, table.getMetaClient().getCommitActionType(),
+ HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS);
+ table.getActiveTimeline().createNewInstant(requested);
+
+ // Setup correct schema and run bulk insert.
+ return getBulkInsertActionExecutor(inputRecordsRDD).execute();
+ }
+
+ protected CommitActionExecutor<T> getBulkInsertActionExecutor(JavaRDD<HoodieRecord> inputRecordsRDD) {
+ return new BulkInsertCommitActionExecutor(jsc, new HoodieWriteConfig.Builder().withProps(config.getProps())
+ .withSchema(bootstrapSchema).build(), table, HoodieTimeline.FULL_BOOTSTRAP_INSTANT_TS,
+ inputRecordsRDD, extraMetadata);
+ }
+
+ private BootstrapWriteStatus handleMetadataBootstrap(String srcPartitionPath, String partitionPath,
+ HoodieFileStatus srcFileStatus, KeyGenerator keyGenerator) {
+
+ Path sourceFilePath = FileStatusUtils.toPath(srcFileStatus.getPath());
+ HoodieBootstrapHandle bootstrapHandle = new HoodieBootstrapHandle(config, HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS,
+ table, partitionPath, FSUtils.createNewFileIdPfx(), table.getSparkTaskContextSupplier());
+ Schema avroSchema = null;
+ try {
+ ParquetMetadata readFooter = ParquetFileReader.readFooter(table.getHadoopConf(), sourceFilePath,
+ ParquetMetadataConverter.NO_FILTER);
+ MessageType parquetSchema = readFooter.getFileMetaData().getSchema();
+ avroSchema = new AvroSchemaConverter().convert(parquetSchema);
+ Schema recordKeySchema = HoodieAvroUtils.generateProjectionSchema(avroSchema,
+ keyGenerator.getTopLevelRecordKeyFields());
+ LOG.info("Schema to be used for reading record Keys :" + recordKeySchema);
+ AvroReadSupport.setAvroReadSchema(table.getHadoopConf(), recordKeySchema);
+ AvroReadSupport.setRequestedProjection(table.getHadoopConf(), recordKeySchema);
+
+ BoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void> wrapper = null;
+ try (ParquetReader<IndexedRecord> reader =
+ AvroParquetReader.<IndexedRecord>builder(sourceFilePath).withConf(table.getHadoopConf()).build()) {
+ wrapper = new SparkBoundedInMemoryExecutor<GenericRecord, HoodieRecord, Void>(config,
+ new ParquetReaderIterator(reader), new BootstrapRecordWriter(bootstrapHandle), inp -> {
+ String recKey = keyGenerator.getRecordKey(inp);
+ GenericRecord gr = new GenericData.Record(HoodieAvroUtils.RECORD_KEY_SCHEMA);
+ gr.put(HoodieRecord.RECORD_KEY_METADATA_FIELD, recKey);
+ BootstrapRecordPayload payload = new BootstrapRecordPayload(gr);
+ HoodieRecord rec = new HoodieRecord(new HoodieKey(recKey, partitionPath), payload);
+ return rec;
+ });
+ wrapper.execute();
+ } catch (Exception e) {
+ throw new HoodieException(e);
+ } finally {
+ bootstrapHandle.close();
+ if (null != wrapper) {
+ wrapper.shutdownNow();
+ }
+ }
+ } catch (IOException e) {
+ throw new HoodieIOException(e.getMessage(), e);
+ }
+ BootstrapWriteStatus writeStatus = (BootstrapWriteStatus)bootstrapHandle.getWriteStatus();
+ BootstrapSourceFileMapping bootstrapSourceFileMapping = new BootstrapSourceFileMapping(
+ config.getBootstrapSourceBasePath(), srcPartitionPath, partitionPath,
+ srcFileStatus, writeStatus.getFileId());
+ writeStatus.setBootstrapSourceFileMapping(bootstrapSourceFileMapping);
+ return writeStatus;
+ }
+
+ /**
+ * Return Bootstrap Mode selections for partitions listed and figure out bootstrap Schema.
+ * @param metaClient Hoodie Table Meta Client.
+ * @return
+ * @throws IOException
+ */
+ private Map<BootstrapMode, List<Pair<String, List<HoodieFileStatus>>>> listAndProcessSourcePartitions(
+ HoodieTableMetaClient metaClient) throws IOException {
+ FileSystem fs = new Path(config.getBootstrapSourceBasePath()).getFileSystem(jsc.hadoopConfiguration());
+ List<Pair<String, List<HoodieFileStatus>>> folders =
+ FSUtils.getAllLeafFoldersWithFiles(bootstrapSourceFileSystem,
+ config.getBootstrapSourceBasePath(), new PathFilter() {
+ @Override
+ public boolean accept(Path path) {
+ // TODO: Needs to be abstracted out when supporting different formats
+ // TODO: Remove hoodieFilter
+ return path.getName().endsWith(".parquet");
Review comment:
Let’s avoid hard coding `.parquet` (Here and any other place in the pr)
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/BootstrapIndex.java
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.ReflectionUtils;
+
+/**
+ * Bootstrap Index Interface.
+ */
+public abstract class BootstrapIndex implements Serializable {
Review comment:
should this actually be an interface? you can then ahve an `AbstractBootstrapIndex` implememting it?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/FileStatusUtils.java
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap;
+
+import org.apache.hudi.avro.model.HoodieFSPermission;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.avro.model.HoodiePath;
+import org.apache.hudi.exception.HoodieIOException;
+
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsAction;
+import org.apache.hadoop.fs.permission.FsPermission;
+
+import java.io.IOException;
+
+/**
+ * Helper functions to convert between FileStatus and HoodieFileStatus.
Review comment:
but in reality this class does more? update teh docs?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/BootstrapIndex.java
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.ReflectionUtils;
+
+/**
+ * Bootstrap Index Interface.
+ */
+public abstract class BootstrapIndex implements Serializable {
+
+ protected static final long serialVersionUID = 1L;
+
+ protected final HoodieTableMetaClient metaClient;
+
+ public BootstrapIndex(HoodieTableMetaClient metaClient) {
+ this.metaClient = metaClient;
+ }
+
+ /**
+ * Create Bootstrap Index Reader.
+ * @return Index Reader
+ */
+ public abstract IndexReader createReader();
+
+ /**
+ * Create Bootstrap Index Writer.
+ * @param sourceBasePath Source Base Path
+ * @return Index Writer
+ */
+ public abstract IndexWriter createWriter(String sourceBasePath);
+
+ /**
+ * Drop bootstrap index.
+ */
+ public abstract void dropIndex();
+
+ /**
+ * Returns true if valid metadata bootstrap is present.
+ * @return
+ */
+ public final boolean isIndexAvailable() {
+ return metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant()
+ .map(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS)).orElse(false) && checkIndex();
+ }
+
+ /**
+ * Check if bootstrap Index is present and ensures readable.
+ */
+ protected abstract boolean checkIndex();
+
+ /**
+ * Bootstrap Index Reader Interface.
+ */
+ public abstract static class IndexReader implements Serializable, AutoCloseable {
+
+ protected final HoodieTableMetaClient metaClient;
+
+ public IndexReader(HoodieTableMetaClient metaClient) {
+ this.metaClient = metaClient;
+ }
+
+ /**
+ * Return bootstrap index info.
+ * @return
+ */
+ public abstract BootstrapIndexInfo getIndexInfo();
+
+ /**
+ * Return Source base path.
+ * @return
+ */
+ public abstract String getSourceBasePath();
+
+ /**
+ * Return list of partitions indexed.
+ * @return
+ */
+ public abstract List<String> getIndexedPartitions();
+
+ /**
+ * Return list file-ids indexed.
+ * @return
+ */
+ public abstract List<String> getIndexedFileIds();
+
+ /**
+ * Lookup bootstrap index by partition.
+ * @param partition Partition to lookup
+ * @return
+ */
+ public abstract List<BootstrapSourceFileMapping> getSourceFileMappingForPartition(String partition);
Review comment:
partitionPath to be consistent with how we use it everywhere
##########
File path: hudi-client/src/main/java/org/apache/hudi/table/action/bootstrap/BootstrapCommitActionExecutor.java
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table.action.bootstrap;
+
+import org.apache.hudi.avro.HoodieAvroUtils;
+import org.apache.hudi.avro.model.HoodieFileStatus;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.client.bootstrap.BootstrapMode;
+import org.apache.hudi.client.bootstrap.BootstrapRecordPayload;
+import org.apache.hudi.client.bootstrap.BootstrapSourceSchemaProvider;
+import org.apache.hudi.client.bootstrap.BootstrapWriteStatus;
+import org.apache.hudi.client.bootstrap.FullBootstrapInputProvider;
+import org.apache.hudi.client.bootstrap.selector.BootstrapModeSelector;
+import org.apache.hudi.client.bootstrap.translator.MetadataBootstrapPartitionPathTranslator;
+import org.apache.hudi.client.utils.ParquetReaderIterator;
+import org.apache.hudi.common.bootstrap.FileStatusUtils;
+import org.apache.hudi.common.bootstrap.index.BootstrapIndex;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieInstant.State;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ReflectionUtils;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.common.util.queue.BoundedInMemoryExecutor;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.execution.SparkBoundedInMemoryExecutor;
+import org.apache.hudi.io.HoodieBootstrapHandle;
+import org.apache.hudi.keygen.KeyGenerator;
+import org.apache.hudi.table.HoodieTable;
+import org.apache.hudi.table.WorkloadProfile;
+import org.apache.hudi.table.action.HoodieWriteMetadata;
+import org.apache.hudi.table.action.commit.BaseCommitActionExecutor;
+import org.apache.hudi.table.action.commit.BulkInsertCommitActionExecutor;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.avro.generic.IndexedRecord;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+import org.apache.hudi.table.action.commit.CommitActionExecutor;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.parquet.avro.AvroParquetReader;
+import org.apache.parquet.avro.AvroReadSupport;
+import org.apache.parquet.avro.AvroSchemaConverter;
+import org.apache.parquet.format.converter.ParquetMetadataConverter;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.schema.MessageType;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class BootstrapCommitActionExecutor<T extends HoodieRecordPayload<T>>
+ extends BaseCommitActionExecutor<T, HoodieBootstrapWriteMetadata> {
+
+ private static final Logger LOG = LogManager.getLogger(BootstrapCommitActionExecutor.class);
+ protected String bootstrapSchema = null;
+ private transient FileSystem bootstrapSourceFileSystem;
+
+ public BootstrapCommitActionExecutor(JavaSparkContext jsc, HoodieWriteConfig config, HoodieTable<?> table,
Review comment:
do all the new ActionExecutor classes have unit tests?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/BootstrapIndex.java
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.ReflectionUtils;
+
+/**
+ * Bootstrap Index Interface.
+ */
+public abstract class BootstrapIndex implements Serializable {
+
+ protected static final long serialVersionUID = 1L;
+
+ protected final HoodieTableMetaClient metaClient;
+
+ public BootstrapIndex(HoodieTableMetaClient metaClient) {
+ this.metaClient = metaClient;
+ }
+
+ /**
+ * Create Bootstrap Index Reader.
+ * @return Index Reader
+ */
+ public abstract IndexReader createReader();
+
+ /**
+ * Create Bootstrap Index Writer.
+ * @param sourceBasePath Source Base Path
+ * @return Index Writer
+ */
+ public abstract IndexWriter createWriter(String sourceBasePath);
+
+ /**
+ * Drop bootstrap index.
+ */
+ public abstract void dropIndex();
+
+ /**
+ * Returns true if valid metadata bootstrap is present.
+ * @return
+ */
+ public final boolean isIndexAvailable() {
+ return metaClient.getActiveTimeline().getCommitsTimeline().filterCompletedInstants().lastInstant()
+ .map(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.GREATER_THAN_OR_EQUALS,
+ HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS)).orElse(false) && checkIndex();
+ }
+
+ /**
+ * Check if bootstrap Index is present and ensures readable.
+ */
+ protected abstract boolean checkIndex();
Review comment:
rename `isIndexPresent()`
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBasedBootstrapIndex.java
##########
@@ -0,0 +1,537 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.bootstrap.index;
+
+import org.apache.hudi.avro.model.BootstrapIndexInfo;
+import org.apache.hudi.avro.model.BootstrapPartitionMetadata;
+import org.apache.hudi.avro.model.BootstrapSourceFilePartitionInfo;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.BootstrapSourceFileMapping;
+import org.apache.hudi.common.model.HoodieFileGroupId;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.io.hfile.CacheConfig;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
+import org.apache.hadoop.hbase.io.hfile.HFileScanner;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Maintains mapping from hudi file id (which contains skeleton file) to external base file.
+ * It maintains 2 physical indices.
+ * (a) At partition granularity to lookup all indices for each partition.
+ * (b) At file-group granularity to lookup bootstrap mapping for an individual file-group.
+ *
+ * This implementation uses HFile as physical storage of index. FOr the initial run, bootstrap
+ * mapping for the entire dataset resides in a single file but care has been taken in naming
+ * the index files in the same way as Hudi data files so that we can reuse file-system abstraction
+ * on these index files to manage multiple file-groups.
+ */
+
+public class HFileBasedBootstrapIndex extends BootstrapIndex {
+
+ protected static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LogManager.getLogger(HFileBasedBootstrapIndex.class);
+
+ public static final String BOOTSTRAP_INDEX_FILE_ID = "00000000-0000-0000-0000-000000000000-0";
Review comment:
are we making any assumptions based on this string? would it still work for arbitrary timestamps eg. ones with millsecond granularity
##########
File path: hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroUtils.java
##########
@@ -326,4 +344,105 @@ public static String decompress(byte[] bytes) {
throw new HoodieIOException("IOException while decompressing text", e);
}
}
+
+ /**
+ * Generate a reader schema off the provided writeSchema, to just project out the provided columns.
+ */
+ public static Schema generateProjectionSchema(Schema originalSchema, List<String> fieldNames) {
+ Map<String, Field> schemaFieldsMap = originalSchema.getFields().stream()
+ .map(r -> Pair.of(r.name().toLowerCase(), r)).collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
+ List<Schema.Field> projectedFields = new ArrayList<>();
+ for (String fn : fieldNames) {
+ Schema.Field field = schemaFieldsMap.get(fn.toLowerCase());
+ if (field == null) {
+ throw new HoodieException("Field " + fn + " not found in log schema. Query cannot proceed! "
+ + "Derived Schema Fields: " + new ArrayList<>(schemaFieldsMap.keySet()));
+ } else {
+ projectedFields.add(new Schema.Field(field.name(), field.schema(), field.doc(), field.defaultValue()));
+ }
+ }
+
+ Schema projectedSchema = Schema.createRecord(originalSchema.getName(), originalSchema.getDoc(),
+ originalSchema.getNamespace(), originalSchema.isError());
+ projectedSchema.setFields(projectedFields);
+ return projectedSchema;
+ }
+
+ /**
+ * Obtain value of the provided field as string, denoted by dot notation. e.g: a.b.c
+ */
+ public static String getNestedFieldValAsString(GenericRecord record, String fieldName, boolean returnNullIfNotFound) {
+ Object obj = getNestedFieldVal(record, fieldName, returnNullIfNotFound);
+ return (obj == null) ? null : obj.toString();
Review comment:
is nt there a StringUtils method for this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #1678: [WIP] [HUDI-242] Metadata Bootstrap changes
Posted by GitBox <gi...@apache.org>.
bvaradar commented on pull request #1678:
URL: https://github.com/apache/hudi/pull/1678#issuecomment-636315006
@umehrot2 @vinothchandar : Unit tests should be passing now. I will add notes to explain high level changes and also annotate part of code myself to describe the reason behind the change where I see fit and ping.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org