You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by GitBox <gi...@apache.org> on 2022/09/13 07:06:58 UTC
[GitHub] [gobblin] meethngala opened a new pull request, #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
meethngala opened a new pull request, #3560:
URL: https://github.com/apache/gobblin/pull/3560
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1709
### Description
- [ ] Here are some details about my PR, including screenshots (if applicable):
- [ ] This PR is focused on Hive Catalog based Iceberg Tables
- [ ] It finds the relevant Iceberg datasets mapping to an Iceberg table and creates copy entities out of all the files that needs to be duplicated as part of the Distcp process
- [ ] Each copy entity is then submitted to the copy source for its respective work unit generation which will then publish the data on the destination
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
- [ ] I have added unit test for finding the relevant Iceberg datasets and generating copy entities out of the exact files to be copied.
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972148398
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergFileSet.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import lombok.Getter;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.partition.FileSet;
+
+/**
+ * A {@link FileSet} for Iceberg datasets. Contains information on Iceberg Dataset associated with an Iceberg table.
+ */
+@Getter
+public abstract class IcebergFileSet extends FileSet<CopyEntity> {
Review Comment:
Now that I think for this PR it might be okay for us to get away with the thin abstraction. I have updated the code based on your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972531227
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -53,37 +54,39 @@ public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDatase
public List<IcebergDataset> findDatasets() throws IOException {
List<IcebergDataset> matchingDatasets = new ArrayList<>();
/*
- * Both Iceberg database name and table name are mandatory,
- * since we are currently only supporting Hive Catalog based Iceberg tables.
- * The design will support defaults and other catalogs in future releases.
+ * Both Iceberg database name and table name are mandatory based on current implementation.
Review Comment:
added it to javadoc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
phet commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973289743
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
Review Comment:
formatting off--need space
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
Review Comment:
where's the `StorageDescriptor` in `Map<Path, FileStatus>`? is this comment correct?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
Review Comment:
if we really think this level of debugging is necessary (I'm currently an even 50-50 split on whether or not...) we could keep it as a `.debug` level, with the idea that it gets selectively enabled, case-by-case
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
Review Comment:
are any of these expected to change or can they be `final`?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
Review Comment:
this feels way too noisy. for reference for each hive daily tracking partition we have > 3600 files! this means that copying 3 partitions would log over 10k lines (once their table management eventually moves from hive to iceberg).
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
Review Comment:
nit: map of paths to what? clearer would be `pathToFileStatus`
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergTableFileSet.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.Collection;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.partition.FileSet;
+
+
+/**
+ * A {@link IcebergTableFileSet} that generates {@link CopyEntity}s for a Hive Catalog based Iceberg table
Review Comment:
I see nothing related to hive catalogs in this impl. In fact, I see nothing related to any kind of iceberg catalog! what felt important to capture here? IMO it risks misleading...
also, no need to repeat the class's own name at the beginning of its javadoc.
let's strike this sentence in favor of the one just below
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePaths() throws IOException {
Review Comment:
not: `getFilePathsToFileStatus`
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePaths() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("Fetching all file paths for the current snapshot id: {} of the Iceberg table with metadata path: {}",
Review Comment:
I like the spirit of the logging, but see room for improvement:
a. be less conversational and more direct, even "mechanical"
b. that's because we want to grep these or otherwise machine-process log lines. hence, strings, especially paths, deserve delimiters to help us detect whitespace (here I used `'`).
c. most of all, include all relevant context in the same message (here, the FQ table ID)
```
log.info("{}.{} - loaded snapshot '{}' from metadata path: '{}'", dbName, tableName, snapId(), mdPath())
```
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = DatasetConstants.PLATFORM_ICEBERG + ".dataset";
+ public static final String ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = ICEBERG_DATASET_PREFIX + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = ICEBERG_DATASET_PREFIX + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ /**
+ * Finds all {@link IcebergDataset}s in the file system using the Iceberg Catalog.
+ * @return List of {@link IcebergDataset}s in the file system.
+ * @throws IOException
+ */
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory based on current implementation.
Review Comment:
better to identify which props are mandatory in the method javadoc, rather than buried here. we aspire for users to learn what they need to know by reading docs, not hunting around in our impl
##########
gobblin-data-management/src/test/java/org/apache/gobblin/runtime/embedded/EmbeddedGobblinDistcpTest.java:
##########
@@ -386,4 +386,4 @@ public Set<Characteristic> applicableCharacteristics() {
}
}
-}
+}
Review Comment:
it's still showing up, even now. let's just remove this file from the commit, since it appears you have nothing to change here anyway
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
Review Comment:
let's think ahead to delta calculation... this should be all paths that exist in the snapshot. the caller will determine what actually "needs to be copied"
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
Review Comment:
if the intent of logging here is to contrast the number of possible paths with the number we determine actually need copying given the delta between source and dest, that's a good call. if so leave a line here, just more information-dense:
```
log.info("{}.{} - found {} candidate source paths", dbName, tableName, mapOfPathsToCopy.size())
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
Review Comment:
what do you feel this line gives (here specifically)?
more typical would be to log the count of `CopyEntity` at the end, just before returning, around line 149.
as always, context is paramount to logging! in this situation, include the db and table name.
finally try to be more terse, less conversational, e.g.:
```
log.info("{}.{} - generated {} copy entities", dbName, tableName, copyEntities.size())
```
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+
+
+public class
+IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_LIST_PATH = "/root/iceberg/test/metadata/test_manifest/data";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to generate copy entities for all the file paths for a mocked iceberg table.
+ * The assumption here is that we create copy entities for all the matching file paths,
+ * without calculating any difference between the source and destination
+ */
+ @Test
+ public void testGenerateCopyEntitiesWhenDestEmpty() throws IOException, URISyntaxException {
+
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ String test_db_name = "test_db_name";
+ String test_table_name = "test_tbl_name";
+ String test_qualified_path = "/root/iceberg/test/destination/sub_path_destination";
+ String test_uri_path = "/root/iceberg/test/output";
+ Properties properties = new Properties();
+ properties.setProperty("data.publisher.final.dir", "/test");
+ List<String> expected = new ArrayList<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_LIST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
+
+ CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
+ .preserve(PreserveAttributes.fromMnemonicString(""))
+ .copyContext(new CopyContext())
+ .build();
+
+ List<String> listedManifestFilePaths = Arrays.asList(MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2);
+ IcebergSnapshotInfo.ManifestFileInfo manifestFileInfo = new IcebergSnapshotInfo.ManifestFileInfo(MANIFEST_LIST_PATH, listedManifestFilePaths);
+ List<IcebergSnapshotInfo.ManifestFileInfo> manifestFiles = Arrays.asList(manifestFileInfo);
+ IcebergTable icebergTable = new MockedIcebergTable(METADATA_PATH, MANIFEST_PATH, manifestFiles);
+ IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+ MockDestinationFileSystem mockDestinationFileSystem = new MockDestinationFileSystem();
+ mockDestinationFileSystem.addPath(METADATA_PATH);
+ mockDestinationFileSystem.addPath(MANIFEST_PATH);
+ mockDestinationFileSystem.addPath(MANIFEST_LIST_PATH);
+ mockDestinationFileSystem.addPath(MANIFEST_FILE_PATH1);
+ mockDestinationFileSystem.addPath(MANIFEST_FILE_PATH2);
+
+ mockFileSystemMethodCalls(fs, mockDestinationFileSystem.pathToFileStatus, test_qualified_path, test_uri_path);
+
+ Collection<CopyEntity> copyEntities = icebergDataset.generateCopyEntities(copyConfiguration);
+ verifyCopyEntities(copyEntities, expected);
+
+ }
+
+ private void verifyCopyEntities(Collection<CopyEntity> copyEntities, List<String> expected) {
+ List<String> actual = new ArrayList<>();
+ for (CopyEntity copyEntity : copyEntities) {
+ String json = copyEntity.toString();
+ JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
+ JsonObject objectData =
+ jsonObject.getAsJsonObject("object-data").getAsJsonObject("origin").getAsJsonObject("object-data");
+ JsonObject pathObject = objectData.getAsJsonObject("path").getAsJsonObject("object-data").getAsJsonObject("uri");
+ String filepath = pathObject.getAsJsonPrimitive("object-data").getAsString();
+ actual.add(filepath);
+ }
+ Assert.assertEquals(actual.size(), expected.size());
+ Assert.assertEqualsNoOrder(actual.toArray(), expected.toArray());
+ }
+
+ private void mockFileSystemMethodCalls(FileSystem fs, Map<Path, FileStatus> pathToFileStatus, String qualifiedPath, String uriPath)
+ throws URISyntaxException, IOException {
+
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, uriPath, null));
+ for (Map.Entry<Path, FileStatus> entry : pathToFileStatus.entrySet()) {
+ Path path = entry.getKey();
+ FileStatus fileStatus = entry.getValue();
+ Mockito.when(fs.getFileStatus(path)).thenReturn(fileStatus);
+ Mockito.when(fs.makeQualified(path)).thenReturn(new Path(qualifiedPath));
+ }
+ }
+
+ private static class MockedIcebergTable extends IcebergTable {
+
+ String metadataPath;
+ String manifestListPath;
+ List<IcebergSnapshotInfo.ManifestFileInfo> manifestFiles;
+
+ public MockedIcebergTable(String metadataPath, String manifestListPath, List<IcebergSnapshotInfo.ManifestFileInfo> manifestFiles) {
+ super(null);
+ this.metadataPath = metadataPath;
+ this.manifestListPath = manifestListPath;
+ this.manifestFiles = manifestFiles;
+ }
+
+ @Override
+ public IcebergSnapshotInfo getCurrentSnapshotInfo() {
+ Long snapshotId = 0L;
+ Instant timestamp = Instant.ofEpochMilli(0L);
+ return new IcebergSnapshotInfo(snapshotId, timestamp, metadataPath, manifestListPath, manifestFiles);
+ }
+ }
+
+ private static class MockDestinationFileSystem {
+ Map<Path, FileStatus> pathToFileStatus;
Review Comment:
this isn't actually itself a mock, so the name is misleading. while you could encapsulate the entirety of the mocking into this class, it's also fine not to...
when you don't go all the way, please just name it appropriately. e.g. `DestinationFileSystemPaths`.
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(hadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePaths() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("Fetching all file paths for the current snapshot of the Iceberg table");
+ List<String> pathsToCopy = icebergSnapshotInfo.getAllPaths();
+
+ for(String pathString : pathsToCopy) {
+ Path path = new Path(pathString);
+ result.put(path, this.fs.getFileStatus(path));
Review Comment:
perhaps will is asking whether a single FNFE or similar should unwind the entire stack and fail us overall, or whether some could be tolerated and we continue on.
there's probably a combo, where some individual failures should fail overall, but others are tolerable--or would even lead us back to reread a fresh `IcebergSnapshotInfo`. we'll refine as we go, so it's ok as you say currently to fail overall for 100% of the cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972193969
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ */
+ @Test
+ public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
+
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ String test_db_name = "test_db_name";
+ String test_table_name = "test_tbl_name";
+ Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
+
+ Properties properties = new Properties();
+ properties.setProperty("data.publisher.final.dir", "/test");
+
+ CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
+ .preserve(PreserveAttributes.fromMnemonicString(""))
+ .copyContext(new CopyContext())
+ .build();
+ TableOperations tableOperations = Mockito.mock(TableOperations.class);
+
+ IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+
+ FileStatus fileStatus1 = new FileStatus();
+ fileStatus1.setPath(new Path(METADATA_PATH));
+ FileStatus fileStatus2 = new FileStatus();
+ fileStatus2.setPath(new Path(MANIFEST_PATH));
+ FileStatus fileStatus3 = new FileStatus();
+ fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ FileStatus fileStatus4 = new FileStatus();
+ fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
+
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_FILE_PATH1);
+ Path path4 = new Path(MANIFEST_FILE_PATH2);
+
+
+ Mockito.when(fs.makeQualified(any(Path.class))).thenReturn(new Path("/root/iceberg/test/destination/sub_path_destination"));
+ Mockito.when(fs.getFileStatus(path1)).thenReturn(fileStatus1);
+ Mockito.when(fs.getFileStatus(path2)).thenReturn(fileStatus2);
+ Mockito.when(fs.getFileStatus(path3)).thenReturn(fileStatus3);
+ Mockito.when(fs.getFileStatus(path4)).thenReturn(fileStatus4);
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, "/root/iceberg/test/output", null));
+
+ Collection<CopyEntity> copyEntities = icebergDataset.generateCopyEntitiesForTableFileSet(copyConfiguration);
+ Assert.assertEquals(copyEntities.size(), setOfFilePaths.size());
+ for (CopyEntity copyEntity : copyEntities) {
+ String json = copyEntity.toString();
+ JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
+ JsonObject objectData =
+ jsonObject.getAsJsonObject("object-data").getAsJsonObject("origin").getAsJsonObject("object-data");
+ JsonObject pathObject = objectData.getAsJsonObject("path").getAsJsonObject("object-data").getAsJsonObject("uri");
+ String filepath = pathObject.getAsJsonPrimitive("object-data").getAsString();
+ Assert.assertTrue(setOfFilePaths.contains(filepath));
+ setOfFilePaths.remove(filepath);
+ }
+ Assert.assertTrue(setOfFilePaths.isEmpty());
+ }
+
+ private static class MockedIcebergTable extends IcebergTable {
+
+ public MockedIcebergTable(TableOperations tableOps) {
+ super(tableOps);
Review Comment:
roger that.... updated the constructor as per suggestions in my latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972187233
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
Review Comment:
good thought... I have updated the method definition and created a helper out of it as part of my latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
Will-Lo commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972726137
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang.StringUtils;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
Review Comment:
We follow an import guideline in Gobblin Intellij (sorry we don't document this well, it was news to me too recently but trying to enforce this where it makes sense).
https://gobblin.readthedocs.io/en/latest/developer-guide/CodingStyle/
You can install this in intellij and have it autoapply in your files
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang.StringUtils;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = DatasetConstants.PLATFORM_ICEBERG + ".dataset";
+ public static final String ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = ICEBERG_DATASET_PREFIX + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = ICEBERG_DATASET_PREFIX + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ /**
+ * Finds all {@link IcebergDataset}s in the file system using the Iceberg Catalog.
+ * @return List of {@link IcebergDataset}s in the file system.
+ * @throws IOException
+ */
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory based on current implementation.
+ * Later we may explore supporting datasets similar to Hive
+ */
+ if (StringUtils.isNotBlank(properties.getProperty(ICEBERG_DB_NAME)) || StringUtils.isNotBlank(properties.getProperty(ICEBERG_TABLE_NAME))) {
+ throw new IllegalArgumentException(String.format("Iceberg database name: {%s} or Iceberg table name: {%s} is missing",
+ ICEBERG_DB_NAME, ICEBERG_TABLE_NAME));
+ }
+ this.dbName = properties.getProperty(ICEBERG_DB_NAME);
+ this.tblName = properties.getProperty(ICEBERG_TABLE_NAME);
+
+ Configuration configuration = HadoopUtils.getConfFromProperties(properties);
+
+ IcebergCatalog icebergCatalog = IcebergCatalogFactory.create(configuration);
+ /* Each Iceberg dataset maps to an Iceberg table
+ * TODO: The user provided database and table names needs to be pre-checked and verified against the existence of a valid Iceberg table
+ */
+ matchingDatasets.add(createIcebergDataset(dbName, tblName, icebergCatalog, properties, fs));
+ log.info("Found {} matching datasets: {}", matchingDatasets.size(), matchingDatasets);
Review Comment:
I like this log :) Can we add the db/table name to it as well?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
Review Comment:
We could name this as defaultHadoopConfiguration just so that the expectation when reading this var is that this is the default configuration on the environment the job is running in, with no modifications from user parameters.
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(hadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePaths() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("Fetching all file paths for the current snapshot of the Iceberg table");
+ List<String> pathsToCopy = icebergSnapshotInfo.getAllPaths();
+
+ for(String pathString : pathsToCopy) {
+ Path path = new Path(pathString);
+ result.put(path, this.fs.getFileStatus(path));
Review Comment:
is there ever a possibility that we run into a FNFE here? I'm thinking if a file gets deleted during the copy by another job, is that possible?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
Review Comment:
This log seems somewhat redundant
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(hadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePaths() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("Fetching all file paths for the current snapshot of the Iceberg table");
Review Comment:
This log can contain more context. i.e., can we include the table uri?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
Review Comment:
I think we can also add more context for this line. I think it's a good practice to imagine that we're trying to debug pipelines so we want to know what tables/files that we're potentially copying and where the error starts, instead of focusing too hard on checkpointing the steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973892214
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
Review Comment:
My intention for logging here was to contrast the number of possible paths that you rightly pointed out. Having said that, I have taken your suggestion and added the log statement as part of my latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973873779
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
Review Comment:
I had named it as a map of paths to be copied over to destination. But, we can refer it as `pathToFileStatus` as well. Updated it in my latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r975581617
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private final IcebergTable icebergTable;
+ protected final Properties properties;
+ protected final FileSystem fs;
Review Comment:
thanks @Will-Lo and @phet for pointing this out. I have renamed the Fs to be more specific... which sourceFs in our case. Update the var in my latest commit!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972165628
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
Review Comment:
yeah I agree... have gotten rid of it in my latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973883221
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePaths() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("Fetching all file paths for the current snapshot id: {} of the Iceberg table with metadata path: {}",
Review Comment:
I agree with the suggestions provided in the spirit of following better logging standards. I have taken your suggestion and included it in my latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973882594
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
Review Comment:
makes sense.... I agree that `getAllPaths()` will determine the paths to be copied for the Iceberg table's current snapshot. I have updated the javadoc accordingly and kept it less verbose in terms of background details for files/paths to be copied!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
phet commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973294122
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
Review Comment:
let's think ahead to delta calculation... this should be all paths that exist in the snapshot. the caller will determine what actually "needs to be copied"
also, don't go so deep on background details in a method's javadoc. maybe just say "Finds all files of the iceberg's current snapshot". I mean, to be clear, you're not even individually working with those files, but just happen to believe they would be among those returned by `getAllPaths()`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
phet commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973294122
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
Review Comment:
let's think ahead to delta calculation... this should be all paths that exist in the snapshot. the caller will determine what actually "needs to be copied"
also, don't go so deep on background details in a method's javadoc. enough to say, "Finds all files of the iceberg's current snapshot". I mean, to be clear, you're not even individually working with those files, but just believe they would be among those returned by `getAllPaths()`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
phet commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r975553754
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private final IcebergTable icebergTable;
+ protected final Properties properties;
+ protected final FileSystem fs;
Review Comment:
good call--definitely helpful! the way it's used in `getFilePathsToFileStatus()` seems like the source FS. please clarify @meethngala
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = DatasetConstants.PLATFORM_ICEBERG + ".dataset";
+ public static final String ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = ICEBERG_DATASET_PREFIX + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = ICEBERG_DATASET_PREFIX + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
Review Comment:
+100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973206997
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang.StringUtils;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = DatasetConstants.PLATFORM_ICEBERG + ".dataset";
+ public static final String ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = ICEBERG_DATASET_PREFIX + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = ICEBERG_DATASET_PREFIX + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ /**
+ * Finds all {@link IcebergDataset}s in the file system using the Iceberg Catalog.
+ * @return List of {@link IcebergDataset}s in the file system.
+ * @throws IOException
+ */
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory based on current implementation.
+ * Later we may explore supporting datasets similar to Hive
+ */
+ if (StringUtils.isNotBlank(properties.getProperty(ICEBERG_DB_NAME)) || StringUtils.isNotBlank(properties.getProperty(ICEBERG_TABLE_NAME))) {
+ throw new IllegalArgumentException(String.format("Iceberg database name: {%s} or Iceberg table name: {%s} is missing",
+ ICEBERG_DB_NAME, ICEBERG_TABLE_NAME));
+ }
+ this.dbName = properties.getProperty(ICEBERG_DB_NAME);
+ this.tblName = properties.getProperty(ICEBERG_TABLE_NAME);
+
+ Configuration configuration = HadoopUtils.getConfFromProperties(properties);
+
+ IcebergCatalog icebergCatalog = IcebergCatalogFactory.create(configuration);
+ /* Each Iceberg dataset maps to an Iceberg table
+ * TODO: The user provided database and table names needs to be pre-checked and verified against the existence of a valid Iceberg table
+ */
+ matchingDatasets.add(createIcebergDataset(dbName, tblName, icebergCatalog, properties, fs));
+ log.info("Found {} matching datasets: {}", matchingDatasets.size(), matchingDatasets);
Review Comment:
done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973494841
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergTableFileSet.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.Collection;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.partition.FileSet;
+
+
+/**
+ * A {@link IcebergTableFileSet} that generates {@link CopyEntity}s for a Hive Catalog based Iceberg table
Review Comment:
I agree Kip... have taken your suggestion to strike off the sentence in the javadoc and kept the below line only in my latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972168238
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
Review Comment:
Added the TODO comment as part of latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972158077
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
+ * The design will support defaults and other catalogs in future releases.
+ */
+ if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
Review Comment:
roger that! Updated the code accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973893265
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
Review Comment:
Honestly, I foresee it to become noisy and hence decided to just keep the overall count of copy entities as part of the log statement in the end which you have pointed out above. The log level as `debug` might be unnecessary for this implementation... and we can always add it later and activate it on a case-by case basis... that's my take on it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972157146
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
Review Comment:
gotcha!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973247681
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.commons.lang.StringUtils;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
Review Comment:
gotcha... addressed it in my latest commit!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r975582202
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private final IcebergTable icebergTable;
+ protected final Properties properties;
+ protected final FileSystem fs;
+
+ private final Optional<String> sourceMetastoreURI;
+ private final Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> pathToFileStatus = getFilePathsToFileStatus();
+ log.info("{}.{} - found {} candidate source paths", dbName, inputTableName, pathToFileStatus.size());
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(pathToFileStatus, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ log.info("{}.{} - generated {} copy entities", dbName, inputTableName, copyEntities.size());
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file path
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> pathToFileStatus, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for (Map.Entry<Path, FileStatus> entry : pathToFileStatus.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for (SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO: Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files of the Iceberg's current snapshot
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePathsToFileStatus() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("{}.{} - loaded snapshot '{}' from metadata path: '{}'", dbName, inputTableName,
+ icebergSnapshotInfo.getSnapshotId(), icebergSnapshotInfo.getMetadataPath());
+ List<String> pathsToCopy = icebergSnapshotInfo.getAllPaths();
+
+ for (String pathString : pathsToCopy) {
+ Path path = new Path(pathString);
+ result.put(path, this.fs.getFileStatus(path));
+ }
+ return result;
+ }
+
+ DatasetDescriptor getSourceDataset() {
+ return getDatasetDescriptor(sourceMetastoreURI);
+ }
+
+ DatasetDescriptor getDestinationDataset() {
+ return getDatasetDescriptor(targetMetastoreURI);
+ }
+
+ @NotNull
+ private DatasetDescriptor getDatasetDescriptor(Optional<String> stringMetastoreURI) {
+ String destinationTable = this.getDbName() + "." + this.getInputTableName();
+
+ URI hiveMetastoreURI = null;
+ if (stringMetastoreURI.isPresent()) {
+ hiveMetastoreURI = URI.create(stringMetastoreURI.get());
+ }
Review Comment:
sure... converted it into a ternary operation and pushed the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973868803
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
Review Comment:
As part of the latest refactoring that I have done... they are not subject to change..... they can be `final`. I have updated the file to incorporate this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r977068365
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -40,18 +39,20 @@
* and creates a {@link IcebergDataset} for each one.
*/
@Slf4j
-@AllArgsConstructor
public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
public static final String ICEBERG_DATASET_PREFIX = DatasetConstants.PLATFORM_ICEBERG + ".dataset";
public static final String ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
public static final String ICEBERG_DB_NAME = ICEBERG_DATASET_PREFIX + ".database.name";
public static final String ICEBERG_TABLE_NAME = ICEBERG_DATASET_PREFIX + ".table.name";
- private String dbName;
- private String tblName;
private final Properties properties;
- protected final FileSystem fs;
+ protected final FileSystem sourceFs;
+
+ public IcebergDatasetFinder(FileSystem fs, Properties properties) {
Review Comment:
updated it to the lombok annotation :)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972160033
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
+ * The design will support defaults and other catalogs in future releases.
+ */
+ if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
+ throw new IOException("Iceberg database name or Iceberg table name is missing");
Review Comment:
yeah.... updated the exception along with string literals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973212838
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
Review Comment:
totally agree. I have added some context the log outlining the files to be copied and where it is getting copied!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972189625
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ */
+ @Test
+ public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
+
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ String test_db_name = "test_db_name";
+ String test_table_name = "test_tbl_name";
+ Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
+
+ Properties properties = new Properties();
+ properties.setProperty("data.publisher.final.dir", "/test");
+
+ CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
+ .preserve(PreserveAttributes.fromMnemonicString(""))
+ .copyContext(new CopyContext())
+ .build();
+ TableOperations tableOperations = Mockito.mock(TableOperations.class);
+
+ IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+
+ FileStatus fileStatus1 = new FileStatus();
+ fileStatus1.setPath(new Path(METADATA_PATH));
+ FileStatus fileStatus2 = new FileStatus();
+ fileStatus2.setPath(new Path(MANIFEST_PATH));
+ FileStatus fileStatus3 = new FileStatus();
+ fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ FileStatus fileStatus4 = new FileStatus();
+ fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
+
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_FILE_PATH1);
+ Path path4 = new Path(MANIFEST_FILE_PATH2);
+
+
+ Mockito.when(fs.makeQualified(any(Path.class))).thenReturn(new Path("/root/iceberg/test/destination/sub_path_destination"));
+ Mockito.when(fs.getFileStatus(path1)).thenReturn(fileStatus1);
+ Mockito.when(fs.getFileStatus(path2)).thenReturn(fileStatus2);
+ Mockito.when(fs.getFileStatus(path3)).thenReturn(fileStatus3);
+ Mockito.when(fs.getFileStatus(path4)).thenReturn(fileStatus4);
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, "/root/iceberg/test/output", null));
+
+ Collection<CopyEntity> copyEntities = icebergDataset.generateCopyEntitiesForTableFileSet(copyConfiguration);
+ Assert.assertEquals(copyEntities.size(), setOfFilePaths.size());
+ for (CopyEntity copyEntity : copyEntities) {
+ String json = copyEntity.toString();
+ JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
+ JsonObject objectData =
+ jsonObject.getAsJsonObject("object-data").getAsJsonObject("origin").getAsJsonObject("object-data");
+ JsonObject pathObject = objectData.getAsJsonObject("path").getAsJsonObject("object-data").getAsJsonObject("uri");
+ String filepath = pathObject.getAsJsonPrimitive("object-data").getAsString();
+ Assert.assertTrue(setOfFilePaths.contains(filepath));
+ setOfFilePaths.remove(filepath);
Review Comment:
this makes more sense.... updated the file!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
phet commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r969935673
##########
gobblin-data-management/src/test/java/org/apache/gobblin/runtime/embedded/EmbeddedGobblinDistcpTest.java:
##########
@@ -386,4 +386,4 @@ public Set<Characteristic> applicableCharacteristics() {
}
}
-}
+}
Review Comment:
typically we have add a trailing newline. overall though, this file may have snuck in unintentionally... (?)
##########
gobblin-runtime/src/main/resources/templates/icebergDistcp.template:
##########
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# ====================================================================
+# Job configurations (can be changed)
+# ====================================================================
+
+# General job metadata
+job.group=DistcpIceberg
+job.name=EmbeddedDistcpIcebergTracking
+job.description=Embedded Iceberg-Distcp-ng toy job
+
Review Comment:
unrelatedly you added a unit test for `IcebergDataset`, and that's looking good.
...have you been able to verify that this config actually works? if not, we might reserve it for a follow-on commit later.
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergTableFileSet.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.Collection;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+
+/**
+ * A {@link IcebergFileSet} that generates {@link CopyEntity}s for a Hive Catalog based Iceberg table
+ */
+public class IcebergTableFileSet extends IcebergFileSet{
Review Comment:
nit: missing a space before '{'
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
Review Comment:
I expected `ICEBERG_DATASET_PREFIX` rather than `DatasetConstants.PLATFORM_ICEBERG`
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
Review Comment:
seems potentially problematic--what's this for? won't it result in NPEs when the core methods of the class are invoked?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntitiesForTableFileSet(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(hadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ Map<Path, FileStatus> getFilePaths() throws IOException {
Review Comment:
why should this be package protected? is it `@VisibleForTesting`? shouldn't it be `private` or `protected`--depending on how we envision subclassing?
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ */
+ @Test
+ public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
+
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ String test_db_name = "test_db_name";
+ String test_table_name = "test_tbl_name";
+ Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
+
+ Properties properties = new Properties();
+ properties.setProperty("data.publisher.final.dir", "/test");
+
+ CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
+ .preserve(PreserveAttributes.fromMnemonicString(""))
+ .copyContext(new CopyContext())
+ .build();
+ TableOperations tableOperations = Mockito.mock(TableOperations.class);
+
+ IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+
+ FileStatus fileStatus1 = new FileStatus();
+ fileStatus1.setPath(new Path(METADATA_PATH));
+ FileStatus fileStatus2 = new FileStatus();
+ fileStatus2.setPath(new Path(MANIFEST_PATH));
+ FileStatus fileStatus3 = new FileStatus();
+ fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ FileStatus fileStatus4 = new FileStatus();
+ fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
+
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_FILE_PATH1);
+ Path path4 = new Path(MANIFEST_FILE_PATH2);
+
+
+ Mockito.when(fs.makeQualified(any(Path.class))).thenReturn(new Path("/root/iceberg/test/destination/sub_path_destination"));
+ Mockito.when(fs.getFileStatus(path1)).thenReturn(fileStatus1);
+ Mockito.when(fs.getFileStatus(path2)).thenReturn(fileStatus2);
+ Mockito.when(fs.getFileStatus(path3)).thenReturn(fileStatus3);
+ Mockito.when(fs.getFileStatus(path4)).thenReturn(fileStatus4);
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, "/root/iceberg/test/output", null));
Review Comment:
this mocking of the `FileSystem` also seems worth abstracting into a helper method
...and, indentation looks off on last line
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
Review Comment:
we're not actually copying files, right? ... just calculating the `CopyEntity` collection
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergFileSet.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import lombok.Getter;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.partition.FileSet;
+
+/**
+ * A {@link FileSet} for Iceberg datasets. Contains information on Iceberg Dataset associated with an Iceberg table.
+ */
+@Getter
+public abstract class IcebergFileSet extends FileSet<CopyEntity> {
Review Comment:
I'm uncertain we actually need this thin abstract fileset with iceberg, in the analogous way we have for hive. the latter has both partitioned and unpartitioned hive file sets, so that gives a common base. at first blush, that's not on the iceberg side.
what would you say to combining them so only one iceberg fileset class? perhaps keep the name `IcebergTableFileSet`, just in case we later realize there is in fact a need for `IcebergFileSet`. if so, we wouldn't then need to rename the class already in use and referenced downstream.
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
Review Comment:
there's no fundamental incompatibility (at least IIRC it's possible to scan an iceberg `HiveCatalog`). seems more more for simplicity, that, to begin with, we copy just one iceberg at a time.
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
+ * The design will support defaults and other catalogs in future releases.
+ */
+ if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
Review Comment:
more common pattern is to test `StringUtils.isNotBlank(properties.getProperty(x))`
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
Review Comment:
nit: suggest: `ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY`
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
+ * The design will support defaults and other catalogs in future releases.
+ */
+ if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
+ throw new IOException("Iceberg database name or Iceberg table name is missing");
+ }
+ this.dbName = properties.getProperty(ICEBERG_DB_NAME);
+ this.tblName = properties.getProperty(ICEBERG_TABLE_NAME);
+
+ Configuration configuration = HadoopUtils.getConfFromProperties(properties);
+
+ IcebergCatalog icebergCatalog = IcebergCatalogFactory.create(configuration);
+ IcebergTable icebergTable = icebergCatalog.openTable(dbName, tblName);
+ // Currently, we only support one dataset per iceberg table
+ matchingDatasets.add(createIcebergDataset(dbName, tblName, icebergTable, properties, fs));
+ log.info("Found {} matching datasets: {}", matchingDatasets.size(), matchingDatasets);
+
+ return matchingDatasets;
+ }
+
+ @Override
+ public Path commonDatasetRoot() {
Review Comment:
what are semantics of `null`? `HiveDatasetFinder` itself has:
```
@Override
public Path commonDatasetRoot() {
return new Path("/");
}
```
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
+ * The design will support defaults and other catalogs in future releases.
+ */
+ if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
+ throw new IOException("Iceberg database name or Iceberg table name is missing");
Review Comment:
doesn't really feel like an `IOException`.
somewhat a stretch (since it's a missing arg), but could be `IllegalArgumentException`. otherwise, just fall back to using `RuntimeException`
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
+ * The design will support defaults and other catalogs in future releases.
+ */
+ if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
+ throw new IOException("Iceberg database name or Iceberg table name is missing");
+ }
+ this.dbName = properties.getProperty(ICEBERG_DB_NAME);
+ this.tblName = properties.getProperty(ICEBERG_TABLE_NAME);
+
+ Configuration configuration = HadoopUtils.getConfFromProperties(properties);
+
+ IcebergCatalog icebergCatalog = IcebergCatalogFactory.create(configuration);
+ IcebergTable icebergTable = icebergCatalog.openTable(dbName, tblName);
+ // Currently, we only support one dataset per iceberg table
+ matchingDatasets.add(createIcebergDataset(dbName, tblName, icebergTable, properties, fs));
Review Comment:
perhaps a comment to note that error handling / verification of table existence comes later in `IcebergTable.getCurrentSnapshotInfo`?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
Review Comment:
doesn't look like a URN, at least not FQ. will this work? if not, maybe sprinkle in a `// TODO: verify!`
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ return getCopyEntities(configuration);
Review Comment:
clearly not implemented, since we don't even touch several input params. please add a todo comment acknowledging incompleteness
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntitiesForTableFileSet(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(hadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ Map<Path, FileStatus> getFilePaths() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("Fetching all file paths for the current snapshot of the Iceberg table");
+ List<String> pathsToCopy = icebergSnapshotInfo.getAllPaths();
+
+ for(String pathString : pathsToCopy) {
+ Path path = new Path(pathString);
+ result.put(path, this.fs.getFileStatus(path));
+ }
+ return result;
+ }
+
+ DatasetDescriptor getSourceDataset() {
+ return getDatasetDescriptor(sourceMetastoreURI);
+ }
+
+ DatasetDescriptor getDestinationDataset() {
+ return getDatasetDescriptor(targetMetastoreURI);
+ }
+
+ @NotNull
+ private DatasetDescriptor getDatasetDescriptor(Optional<String> stringMetastoreURI) {
+ String destinationTable = this.getDbName() + "." + this.getInputTableName();
+
+ URI hiveMetastoreURI = null;
+ if (stringMetastoreURI.isPresent()) {
+ hiveMetastoreURI = URI.create(stringMetastoreURI.get());
+ }
+
+ DatasetDescriptor destinationDataset =
+ new DatasetDescriptor(DatasetConstants.PLATFORM_ICEBERG, hiveMetastoreURI, destinationTable);
+ destinationDataset.addMetadata(DatasetConstants.FS_URI, this.getFs().getUri().toString());
Review Comment:
likely to require tweaking eventually, esp. since not all iceberg catalogs are hive metastores. I'm also unclear what it means in cases where the HMS URI is null...
nonetheless, may be fine in the very near term for this particular commit
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
Review Comment:
would be nice to abstract all of this to a helper:
```
IcebergTable mockIcebergTable(pathsToCopy, whateverElse...)
```
we'll likely need more than a single test, and that will help to write each n+1
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
+ * The design will support defaults and other catalogs in future releases.
+ */
+ if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
+ throw new IOException("Iceberg database name or Iceberg table name is missing");
+ }
+ this.dbName = properties.getProperty(ICEBERG_DB_NAME);
+ this.tblName = properties.getProperty(ICEBERG_TABLE_NAME);
+
+ Configuration configuration = HadoopUtils.getConfFromProperties(properties);
+
+ IcebergCatalog icebergCatalog = IcebergCatalogFactory.create(configuration);
+ IcebergTable icebergTable = icebergCatalog.openTable(dbName, tblName);
+ // Currently, we only support one dataset per iceberg table
+ matchingDatasets.add(createIcebergDataset(dbName, tblName, icebergTable, properties, fs));
+ log.info("Found {} matching datasets: {}", matchingDatasets.size(), matchingDatasets);
+
+ return matchingDatasets;
+ }
+
+ @Override
+ public Path commonDatasetRoot() {
+ return null;
+ }
+
+ @Override
+ public Iterator<IcebergDataset> getDatasetsIterator() throws IOException {
+ return findDatasets().iterator();
+ }
+ protected IcebergDataset createIcebergDataset(String dbName, String tblName, IcebergTable icebergTable, Properties properties, FileSystem fs) {
Review Comment:
minor, but you could pass in the `IcebergCatalog` and call `openTable()` within to gain the `IcebergTable` you'll eventually pass to the `IcebergDataset` ctor
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
Review Comment:
since it's just a POJO, I'd expect you to just return a value, rather than make a mock... although perhaps not a big difference either way...
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntitiesForTableFileSet(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
Review Comment:
unsure, but would this be where we'd point to whatever target cluster we're supposed to copy to? I believe we'll eventually need more than empty config...
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
Review Comment:
also, I don't see this being used below (although I do expect we'll eventually need to incorporate it)
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ */
+ @Test
+ public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
+
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ String test_db_name = "test_db_name";
+ String test_table_name = "test_tbl_name";
+ Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
+
+ Properties properties = new Properties();
+ properties.setProperty("data.publisher.final.dir", "/test");
+
+ CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
+ .preserve(PreserveAttributes.fromMnemonicString(""))
+ .copyContext(new CopyContext())
+ .build();
+ TableOperations tableOperations = Mockito.mock(TableOperations.class);
+
+ IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+
+ FileStatus fileStatus1 = new FileStatus();
+ fileStatus1.setPath(new Path(METADATA_PATH));
+ FileStatus fileStatus2 = new FileStatus();
+ fileStatus2.setPath(new Path(MANIFEST_PATH));
+ FileStatus fileStatus3 = new FileStatus();
+ fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ FileStatus fileStatus4 = new FileStatus();
+ fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
+
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_FILE_PATH1);
+ Path path4 = new Path(MANIFEST_FILE_PATH2);
+
+
+ Mockito.when(fs.makeQualified(any(Path.class))).thenReturn(new Path("/root/iceberg/test/destination/sub_path_destination"));
+ Mockito.when(fs.getFileStatus(path1)).thenReturn(fileStatus1);
+ Mockito.when(fs.getFileStatus(path2)).thenReturn(fileStatus2);
+ Mockito.when(fs.getFileStatus(path3)).thenReturn(fileStatus3);
+ Mockito.when(fs.getFileStatus(path4)).thenReturn(fileStatus4);
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, "/root/iceberg/test/output", null));
+
+ Collection<CopyEntity> copyEntities = icebergDataset.generateCopyEntitiesForTableFileSet(copyConfiguration);
+ Assert.assertEquals(copyEntities.size(), setOfFilePaths.size());
+ for (CopyEntity copyEntity : copyEntities) {
+ String json = copyEntity.toString();
+ JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
+ JsonObject objectData =
+ jsonObject.getAsJsonObject("object-data").getAsJsonObject("origin").getAsJsonObject("object-data");
+ JsonObject pathObject = objectData.getAsJsonObject("path").getAsJsonObject("object-data").getAsJsonObject("uri");
+ String filepath = pathObject.getAsJsonPrimitive("object-data").getAsString();
+ Assert.assertTrue(setOfFilePaths.contains(filepath));
+ setOfFilePaths.remove(filepath);
+ }
+ Assert.assertTrue(setOfFilePaths.isEmpty());
Review Comment:
again, let's use a helper. we definitely prefer not to repeat ourselves as we write more tests that similarly wish to validate against expectations.
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ */
+ @Test
+ public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
+
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ String test_db_name = "test_db_name";
+ String test_table_name = "test_tbl_name";
+ Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
+
+ Properties properties = new Properties();
+ properties.setProperty("data.publisher.final.dir", "/test");
+
+ CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
+ .preserve(PreserveAttributes.fromMnemonicString(""))
+ .copyContext(new CopyContext())
+ .build();
+ TableOperations tableOperations = Mockito.mock(TableOperations.class);
+
+ IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+
+ FileStatus fileStatus1 = new FileStatus();
+ fileStatus1.setPath(new Path(METADATA_PATH));
+ FileStatus fileStatus2 = new FileStatus();
+ fileStatus2.setPath(new Path(MANIFEST_PATH));
+ FileStatus fileStatus3 = new FileStatus();
+ fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ FileStatus fileStatus4 = new FileStatus();
+ fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
+
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_FILE_PATH1);
+ Path path4 = new Path(MANIFEST_FILE_PATH2);
+
+
+ Mockito.when(fs.makeQualified(any(Path.class))).thenReturn(new Path("/root/iceberg/test/destination/sub_path_destination"));
+ Mockito.when(fs.getFileStatus(path1)).thenReturn(fileStatus1);
+ Mockito.when(fs.getFileStatus(path2)).thenReturn(fileStatus2);
+ Mockito.when(fs.getFileStatus(path3)).thenReturn(fileStatus3);
+ Mockito.when(fs.getFileStatus(path4)).thenReturn(fileStatus4);
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, "/root/iceberg/test/output", null));
+
+ Collection<CopyEntity> copyEntities = icebergDataset.generateCopyEntitiesForTableFileSet(copyConfiguration);
+ Assert.assertEquals(copyEntities.size(), setOfFilePaths.size());
+ for (CopyEntity copyEntity : copyEntities) {
+ String json = copyEntity.toString();
+ JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
+ JsonObject objectData =
+ jsonObject.getAsJsonObject("object-data").getAsJsonObject("origin").getAsJsonObject("object-data");
+ JsonObject pathObject = objectData.getAsJsonObject("path").getAsJsonObject("object-data").getAsJsonObject("uri");
+ String filepath = pathObject.getAsJsonPrimitive("object-data").getAsString();
+ Assert.assertTrue(setOfFilePaths.contains(filepath));
+ setOfFilePaths.remove(filepath);
+ }
+ Assert.assertTrue(setOfFilePaths.isEmpty());
+ }
+
+ private static class MockedIcebergTable extends IcebergTable {
+
+ public MockedIcebergTable(TableOperations tableOps) {
+ super(tableOps);
Review Comment:
no need for a `TableOperations` param: if you find it won't work to just pass `super(null)`, go ahead and instantiate the mock herein.
instead, let's have params to receive the various paths, so we're able to write more than this one test with its hard-coded values
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ */
+ @Test
+ public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
+
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ String test_db_name = "test_db_name";
+ String test_table_name = "test_tbl_name";
+ Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
+
+ Properties properties = new Properties();
+ properties.setProperty("data.publisher.final.dir", "/test");
+
+ CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
+ .preserve(PreserveAttributes.fromMnemonicString(""))
+ .copyContext(new CopyContext())
+ .build();
+ TableOperations tableOperations = Mockito.mock(TableOperations.class);
+
+ IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+
+ FileStatus fileStatus1 = new FileStatus();
+ fileStatus1.setPath(new Path(METADATA_PATH));
+ FileStatus fileStatus2 = new FileStatus();
+ fileStatus2.setPath(new Path(MANIFEST_PATH));
+ FileStatus fileStatus3 = new FileStatus();
+ fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ FileStatus fileStatus4 = new FileStatus();
+ fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
+
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_FILE_PATH1);
+ Path path4 = new Path(MANIFEST_FILE_PATH2);
+
+
+ Mockito.when(fs.makeQualified(any(Path.class))).thenReturn(new Path("/root/iceberg/test/destination/sub_path_destination"));
+ Mockito.when(fs.getFileStatus(path1)).thenReturn(fileStatus1);
+ Mockito.when(fs.getFileStatus(path2)).thenReturn(fileStatus2);
+ Mockito.when(fs.getFileStatus(path3)).thenReturn(fileStatus3);
+ Mockito.when(fs.getFileStatus(path4)).thenReturn(fileStatus4);
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, "/root/iceberg/test/output", null));
+
+ Collection<CopyEntity> copyEntities = icebergDataset.generateCopyEntitiesForTableFileSet(copyConfiguration);
+ Assert.assertEquals(copyEntities.size(), setOfFilePaths.size());
+ for (CopyEntity copyEntity : copyEntities) {
+ String json = copyEntity.toString();
+ JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
+ JsonObject objectData =
+ jsonObject.getAsJsonObject("object-data").getAsJsonObject("origin").getAsJsonObject("object-data");
+ JsonObject pathObject = objectData.getAsJsonObject("path").getAsJsonObject("object-data").getAsJsonObject("uri");
+ String filepath = pathObject.getAsJsonPrimitive("object-data").getAsString();
+ Assert.assertTrue(setOfFilePaths.contains(filepath));
+ setOfFilePaths.remove(filepath);
Review Comment:
would this help - https://stackoverflow.com/a/47064441 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972188247
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
Review Comment:
that's right... updated my comment conveying the same. I understand it could lead to confusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973894077
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = DatasetConstants.PLATFORM_ICEBERG + ".dataset";
+ public static final String ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = ICEBERG_DATASET_PREFIX + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = ICEBERG_DATASET_PREFIX + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ /**
+ * Finds all {@link IcebergDataset}s in the file system using the Iceberg Catalog.
+ * @return List of {@link IcebergDataset}s in the file system.
+ * @throws IOException
+ */
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory based on current implementation.
Review Comment:
roger that... I have added the information as part of javadoc!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r977068142
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -202,14 +202,11 @@ DatasetDescriptor getDestinationDataset() {
private DatasetDescriptor getDatasetDescriptor(Optional<String> stringMetastoreURI) {
String destinationTable = this.getDbName() + "." + this.getInputTableName();
- URI hiveMetastoreURI = null;
- if (stringMetastoreURI.isPresent()) {
- hiveMetastoreURI = URI.create(stringMetastoreURI.get());
- }
+ URI hiveMetastoreURI = stringMetastoreURI.isPresent() ? URI.create(stringMetastoreURI.get()) : null;
DatasetDescriptor destinationDataset =
new DatasetDescriptor(DatasetConstants.PLATFORM_ICEBERG, hiveMetastoreURI, destinationTable);
- destinationDataset.addMetadata(DatasetConstants.FS_URI, this.getFs().getUri().toString());
+ destinationDataset.addMetadata(DatasetConstants.FS_URI, this.getSourceFs().getUri().toString());
Review Comment:
updated the File System to be as per source and destination in my latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] Will-Lo merged pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
Will-Lo merged PR #3560:
URL: https://github.com/apache/gobblin/pull/3560
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972190380
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ */
+ @Test
+ public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
+
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ String test_db_name = "test_db_name";
+ String test_table_name = "test_tbl_name";
+ Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
+
+ Properties properties = new Properties();
+ properties.setProperty("data.publisher.final.dir", "/test");
+
+ CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
+ .preserve(PreserveAttributes.fromMnemonicString(""))
+ .copyContext(new CopyContext())
+ .build();
+ TableOperations tableOperations = Mockito.mock(TableOperations.class);
+
+ IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+
+ FileStatus fileStatus1 = new FileStatus();
+ fileStatus1.setPath(new Path(METADATA_PATH));
+ FileStatus fileStatus2 = new FileStatus();
+ fileStatus2.setPath(new Path(MANIFEST_PATH));
+ FileStatus fileStatus3 = new FileStatus();
+ fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ FileStatus fileStatus4 = new FileStatus();
+ fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
+
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_FILE_PATH1);
+ Path path4 = new Path(MANIFEST_FILE_PATH2);
+
+
+ Mockito.when(fs.makeQualified(any(Path.class))).thenReturn(new Path("/root/iceberg/test/destination/sub_path_destination"));
+ Mockito.when(fs.getFileStatus(path1)).thenReturn(fileStatus1);
+ Mockito.when(fs.getFileStatus(path2)).thenReturn(fileStatus2);
+ Mockito.when(fs.getFileStatus(path3)).thenReturn(fileStatus3);
+ Mockito.when(fs.getFileStatus(path4)).thenReturn(fileStatus4);
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, "/root/iceberg/test/output", null));
+
+ Collection<CopyEntity> copyEntities = icebergDataset.generateCopyEntitiesForTableFileSet(copyConfiguration);
+ Assert.assertEquals(copyEntities.size(), setOfFilePaths.size());
+ for (CopyEntity copyEntity : copyEntities) {
+ String json = copyEntity.toString();
+ JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
+ JsonObject objectData =
+ jsonObject.getAsJsonObject("object-data").getAsJsonObject("origin").getAsJsonObject("object-data");
+ JsonObject pathObject = objectData.getAsJsonObject("path").getAsJsonObject("object-data").getAsJsonObject("uri");
+ String filepath = pathObject.getAsJsonPrimitive("object-data").getAsString();
+ Assert.assertTrue(setOfFilePaths.contains(filepath));
+ setOfFilePaths.remove(filepath);
+ }
+ Assert.assertTrue(setOfFilePaths.isEmpty());
Review Comment:
yeah for sure.... would not want to repeat it. I have created a helper method for this as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
phet commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972350695
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -53,37 +54,39 @@ public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDatase
public List<IcebergDataset> findDatasets() throws IOException {
List<IcebergDataset> matchingDatasets = new ArrayList<>();
/*
- * Both Iceberg database name and table name are mandatory,
- * since we are currently only supporting Hive Catalog based Iceberg tables.
- * The design will support defaults and other catalogs in future releases.
+ * Both Iceberg database name and table name are mandatory based on current implementation.
Review Comment:
please put into method javadoc, to show up when reading in that tool
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -77,80 +74,112 @@ public void testGetFilePaths() throws IOException {
}
/**
- * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ * Test case to generate copy entities for all the file paths for a mocked iceberg table.
+ * The assumption here is that we create copy entities for all the matching file paths,
+ * without calculating any difference between the source and destination
*/
@Test
public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
Review Comment:
maybe rename so test name tracks to comment above... such as `testGenerateCopyEntitiesWhenDestEmpty`?
(it's not so much we wouldn't calculate a difference between src and destination, but that this initial, simplest test mocks a destination that has no data yet)
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -77,80 +74,112 @@ public void testGetFilePaths() throws IOException {
}
/**
- * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ * Test case to generate copy entities for all the file paths for a mocked iceberg table.
+ * The assumption here is that we create copy entities for all the matching file paths,
+ * without calculating any difference between the source and destination
*/
@Test
public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
FileSystem fs = Mockito.mock(FileSystem.class);
String test_db_name = "test_db_name";
String test_table_name = "test_tbl_name";
- Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
-
+ String test_qualified_path = "/root/iceberg/test/destination/sub_path_destination";
+ String test_uri_path = "/root/iceberg/test/output";
Properties properties = new Properties();
properties.setProperty("data.publisher.final.dir", "/test");
+ List<String> expected = new ArrayList<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_LIST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
.preserve(PreserveAttributes.fromMnemonicString(""))
.copyContext(new CopyContext())
.build();
- TableOperations tableOperations = Mockito.mock(TableOperations.class);
- IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergTable icebergTable = new MockedIcebergTable(METADATA_PATH, MANIFEST_PATH, MANIFEST_LIST_PATH, new ArrayList<>(Arrays.asList(MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2)));
IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_LIST_PATH);
+ Path path4 = new Path(MANIFEST_FILE_PATH1);
+ Path path5 = new Path(MANIFEST_FILE_PATH2);
+
FileStatus fileStatus1 = new FileStatus();
- fileStatus1.setPath(new Path(METADATA_PATH));
+ fileStatus1.setPath(path1);
FileStatus fileStatus2 = new FileStatus();
- fileStatus2.setPath(new Path(MANIFEST_PATH));
+ fileStatus2.setPath(path2);
FileStatus fileStatus3 = new FileStatus();
- fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ fileStatus3.setPath(path3);
FileStatus fileStatus4 = new FileStatus();
- fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
-
- Path path1 = new Path(METADATA_PATH);
- Path path2 = new Path(MANIFEST_PATH);
- Path path3 = new Path(MANIFEST_FILE_PATH1);
- Path path4 = new Path(MANIFEST_FILE_PATH2);
+ fileStatus4.setPath(path4);
+ FileStatus fileStatus5 = new FileStatus();
+ fileStatus5.setPath(path5);
+ Map<Path, FileStatus> mapOfPathAndFileStatus = Maps.newHashMap();
+ mapOfPathAndFileStatus.put(path1, fileStatus1);
+ mapOfPathAndFileStatus.put(path2, fileStatus2);
+ mapOfPathAndFileStatus.put(path3, fileStatus3);
+ mapOfPathAndFileStatus.put(path4, fileStatus4);
+ mapOfPathAndFileStatus.put(path5, fileStatus5);
Review Comment:
these 20+ lines for 5 file paths seems a too much repetition and boiler plate. could we invent an abstraction, like `MockDestinationFileSystem` and give it methods:
```
MockDestinationFileSystem addPath(String p); // empty `FileStatus` (indicating not found)
MockDestinationFileSystem addPath(String p, FileStatus fstat);
```
?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergTableFileSet.java:
##########
@@ -21,20 +21,26 @@
import java.util.Collection;
import org.apache.gobblin.data.management.copy.CopyConfiguration;
import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.partition.FileSet;
+
/**
- * A {@link IcebergFileSet} that generates {@link CopyEntity}s for a Hive Catalog based Iceberg table
+ * A {@link IcebergTableFileSet} that generates {@link CopyEntity}s for a Hive Catalog based Iceberg table
+ * A {@link FileSet} for Iceberg datasets containing information associated with an Iceberg table.
*/
-public class IcebergTableFileSet extends IcebergFileSet{
+public class IcebergTableFileSet extends FileSet<CopyEntity> {
private final CopyConfiguration copyConfiguration;
+ private final IcebergDataset icebergDataset;
+
public IcebergTableFileSet(String name, IcebergDataset icebergDataset, CopyConfiguration configuration) {
super(name, icebergDataset);
this.copyConfiguration = configuration;
+ this.icebergDataset = icebergDataset;
}
@Override
protected Collection<CopyEntity> generateCopyEntities() throws IOException {
- return this.getIcebergDataset().generateCopyEntitiesForTableFileSet(this.copyConfiguration);
+ return this.icebergDataset.generateCopyEntitiesForTableFileSet(this.copyConfiguration);
Review Comment:
more I see this, I'd prefer to shorten the name to `IcebergDataset.generateCopyEntities`. whaddya say?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -53,37 +54,39 @@ public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDatase
public List<IcebergDataset> findDatasets() throws IOException {
List<IcebergDataset> matchingDatasets = new ArrayList<>();
/*
- * Both Iceberg database name and table name are mandatory,
- * since we are currently only supporting Hive Catalog based Iceberg tables.
- * The design will support defaults and other catalogs in future releases.
+ * Both Iceberg database name and table name are mandatory based on current implementation.
+ * Later we may explore supporting datasets similar to Hive
*/
- if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
- throw new IOException("Iceberg database name or Iceberg table name is missing");
+ if (StringUtils.isNotBlank(properties.getProperty(ICEBERG_DB_NAME)) || StringUtils.isNotBlank(properties.getProperty(ICEBERG_TABLE_NAME))) {
+ throw new IllegalArgumentException(String.format("Iceberg database name: {%s} or Iceberg table name: {%s} is missing",
+ ICEBERG_DB_NAME, ICEBERG_TABLE_NAME));
}
this.dbName = properties.getProperty(ICEBERG_DB_NAME);
this.tblName = properties.getProperty(ICEBERG_TABLE_NAME);
Configuration configuration = HadoopUtils.getConfFromProperties(properties);
IcebergCatalog icebergCatalog = IcebergCatalogFactory.create(configuration);
- IcebergTable icebergTable = icebergCatalog.openTable(dbName, tblName);
- // Currently, we only support one dataset per iceberg table
- matchingDatasets.add(createIcebergDataset(dbName, tblName, icebergTable, properties, fs));
+ /* Currently, we only support one dataset per iceberg table
+ * Error handling and verification of table existence will be included as part IcebergTable.getCurrentSnapshotInfo() in future releases.
Review Comment:
not a future release. merely it will happen later on (during this same execution).
thinking more on it, this may be the best time to indicate to the user that the db and table names they gave don't indicate a known iceberg table--kind of a pre-verification. either quickly add or else leave a todo... but seems you could use:
```
Catalog.listNamespaces(dbName)
Catalog.listTables(namespace)
```
to probe for that.
the last return value isn't ideal, and I see no better alternative than to sequentially scan the `List<String>` returned to look for `tblName`
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -77,80 +74,112 @@ public void testGetFilePaths() throws IOException {
}
/**
- * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ * Test case to generate copy entities for all the file paths for a mocked iceberg table.
+ * The assumption here is that we create copy entities for all the matching file paths,
+ * without calculating any difference between the source and destination
*/
@Test
public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
FileSystem fs = Mockito.mock(FileSystem.class);
String test_db_name = "test_db_name";
String test_table_name = "test_tbl_name";
- Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
-
+ String test_qualified_path = "/root/iceberg/test/destination/sub_path_destination";
+ String test_uri_path = "/root/iceberg/test/output";
Properties properties = new Properties();
properties.setProperty("data.publisher.final.dir", "/test");
+ List<String> expected = new ArrayList<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_LIST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
.preserve(PreserveAttributes.fromMnemonicString(""))
.copyContext(new CopyContext())
.build();
- TableOperations tableOperations = Mockito.mock(TableOperations.class);
- IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergTable icebergTable = new MockedIcebergTable(METADATA_PATH, MANIFEST_PATH, MANIFEST_LIST_PATH, new ArrayList<>(Arrays.asList(MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2)));
IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_LIST_PATH);
+ Path path4 = new Path(MANIFEST_FILE_PATH1);
+ Path path5 = new Path(MANIFEST_FILE_PATH2);
+
FileStatus fileStatus1 = new FileStatus();
- fileStatus1.setPath(new Path(METADATA_PATH));
+ fileStatus1.setPath(path1);
FileStatus fileStatus2 = new FileStatus();
- fileStatus2.setPath(new Path(MANIFEST_PATH));
+ fileStatus2.setPath(path2);
FileStatus fileStatus3 = new FileStatus();
- fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ fileStatus3.setPath(path3);
FileStatus fileStatus4 = new FileStatus();
- fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
-
- Path path1 = new Path(METADATA_PATH);
- Path path2 = new Path(MANIFEST_PATH);
- Path path3 = new Path(MANIFEST_FILE_PATH1);
- Path path4 = new Path(MANIFEST_FILE_PATH2);
+ fileStatus4.setPath(path4);
+ FileStatus fileStatus5 = new FileStatus();
+ fileStatus5.setPath(path5);
+ Map<Path, FileStatus> mapOfPathAndFileStatus = Maps.newHashMap();
+ mapOfPathAndFileStatus.put(path1, fileStatus1);
+ mapOfPathAndFileStatus.put(path2, fileStatus2);
+ mapOfPathAndFileStatus.put(path3, fileStatus3);
+ mapOfPathAndFileStatus.put(path4, fileStatus4);
+ mapOfPathAndFileStatus.put(path5, fileStatus5);
- Mockito.when(fs.makeQualified(any(Path.class))).thenReturn(new Path("/root/iceberg/test/destination/sub_path_destination"));
- Mockito.when(fs.getFileStatus(path1)).thenReturn(fileStatus1);
- Mockito.when(fs.getFileStatus(path2)).thenReturn(fileStatus2);
- Mockito.when(fs.getFileStatus(path3)).thenReturn(fileStatus3);
- Mockito.when(fs.getFileStatus(path4)).thenReturn(fileStatus4);
- Mockito.when(fs.getUri()).thenReturn(new URI(null, null, "/root/iceberg/test/output", null));
+ mockFileSystemMethodCalls(fs, mapOfPathAndFileStatus, test_qualified_path, test_uri_path);
Collection<CopyEntity> copyEntities = icebergDataset.generateCopyEntitiesForTableFileSet(copyConfiguration);
- Assert.assertEquals(copyEntities.size(), setOfFilePaths.size());
+ verifyCopyEntities(copyEntities, expected);
+
+ }
+
+ private void verifyCopyEntities(Collection<CopyEntity> copyEntities, List<String> expected) {
+ List<String> actual = new ArrayList<>();
for (CopyEntity copyEntity : copyEntities) {
String json = copyEntity.toString();
JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
JsonObject objectData =
jsonObject.getAsJsonObject("object-data").getAsJsonObject("origin").getAsJsonObject("object-data");
JsonObject pathObject = objectData.getAsJsonObject("path").getAsJsonObject("object-data").getAsJsonObject("uri");
String filepath = pathObject.getAsJsonPrimitive("object-data").getAsString();
- Assert.assertTrue(setOfFilePaths.contains(filepath));
- setOfFilePaths.remove(filepath);
+ actual.add(filepath);
+ }
+ Assert.assertEquals(actual.size(), expected.size());
+ Assert.assertEqualsNoOrder(actual.toArray(), expected.toArray());
+ }
+
+ private void mockFileSystemMethodCalls(FileSystem fs, Map<Path, FileStatus> pathToFileStatus, String qualifiedPath, String uriPath)
+ throws URISyntaxException, IOException {
+
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, uriPath, null));
+ for (Map.Entry<Path, FileStatus> entry : pathToFileStatus.entrySet()) {
+ Path path = entry.getKey();
+ FileStatus fileStatus = entry.getValue();
+ Mockito.when(fs.getFileStatus(path)).thenReturn(fileStatus);
+ Mockito.when(fs.makeQualified(path)).thenReturn(new Path(qualifiedPath));
}
- Assert.assertTrue(setOfFilePaths.isEmpty());
}
private static class MockedIcebergTable extends IcebergTable {
- public MockedIcebergTable(TableOperations tableOps) {
- super(tableOps);
+ String metadataPath;
+ String manifestListPath;
+ String manifestFilePath;
+ List<String> manifestListFilePaths;
Review Comment:
this mocking only works for a single manifest file (e.g. when the manifest list has only one entry). let's generalize to the full model, which is a list of manifest files, each with their own list of listed (data) file paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972531688
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -77,80 +74,112 @@ public void testGetFilePaths() throws IOException {
}
/**
- * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ * Test case to generate copy entities for all the file paths for a mocked iceberg table.
+ * The assumption here is that we create copy entities for all the matching file paths,
+ * without calculating any difference between the source and destination
*/
@Test
public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
FileSystem fs = Mockito.mock(FileSystem.class);
String test_db_name = "test_db_name";
String test_table_name = "test_tbl_name";
- Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
-
+ String test_qualified_path = "/root/iceberg/test/destination/sub_path_destination";
+ String test_uri_path = "/root/iceberg/test/output";
Properties properties = new Properties();
properties.setProperty("data.publisher.final.dir", "/test");
+ List<String> expected = new ArrayList<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_LIST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
.preserve(PreserveAttributes.fromMnemonicString(""))
.copyContext(new CopyContext())
.build();
- TableOperations tableOperations = Mockito.mock(TableOperations.class);
- IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergTable icebergTable = new MockedIcebergTable(METADATA_PATH, MANIFEST_PATH, MANIFEST_LIST_PATH, new ArrayList<>(Arrays.asList(MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2)));
IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_LIST_PATH);
+ Path path4 = new Path(MANIFEST_FILE_PATH1);
+ Path path5 = new Path(MANIFEST_FILE_PATH2);
+
FileStatus fileStatus1 = new FileStatus();
- fileStatus1.setPath(new Path(METADATA_PATH));
+ fileStatus1.setPath(path1);
FileStatus fileStatus2 = new FileStatus();
- fileStatus2.setPath(new Path(MANIFEST_PATH));
+ fileStatus2.setPath(path2);
FileStatus fileStatus3 = new FileStatus();
- fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ fileStatus3.setPath(path3);
FileStatus fileStatus4 = new FileStatus();
- fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
-
- Path path1 = new Path(METADATA_PATH);
- Path path2 = new Path(MANIFEST_PATH);
- Path path3 = new Path(MANIFEST_FILE_PATH1);
- Path path4 = new Path(MANIFEST_FILE_PATH2);
+ fileStatus4.setPath(path4);
+ FileStatus fileStatus5 = new FileStatus();
+ fileStatus5.setPath(path5);
+ Map<Path, FileStatus> mapOfPathAndFileStatus = Maps.newHashMap();
+ mapOfPathAndFileStatus.put(path1, fileStatus1);
+ mapOfPathAndFileStatus.put(path2, fileStatus2);
+ mapOfPathAndFileStatus.put(path3, fileStatus3);
+ mapOfPathAndFileStatus.put(path4, fileStatus4);
+ mapOfPathAndFileStatus.put(path5, fileStatus5);
Review Comment:
that's a good idea... pushed the implementation as part of the latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972531483
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -77,80 +74,112 @@ public void testGetFilePaths() throws IOException {
}
/**
- * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ * Test case to generate copy entities for all the file paths for a mocked iceberg table.
+ * The assumption here is that we create copy entities for all the matching file paths,
+ * without calculating any difference between the source and destination
*/
@Test
public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
Review Comment:
done!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972158731
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
Review Comment:
makes sense. Updated the doc accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972156858
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
Review Comment:
updated the file as per your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r975582599
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = DatasetConstants.PLATFORM_ICEBERG + ".dataset";
+ public static final String ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = ICEBERG_DATASET_PREFIX + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = ICEBERG_DATASET_PREFIX + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
Review Comment:
specified the fs in my latest commit!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
Will-Lo commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r975677535
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -202,14 +202,11 @@ DatasetDescriptor getDestinationDataset() {
private DatasetDescriptor getDatasetDescriptor(Optional<String> stringMetastoreURI) {
String destinationTable = this.getDbName() + "." + this.getInputTableName();
- URI hiveMetastoreURI = null;
- if (stringMetastoreURI.isPresent()) {
- hiveMetastoreURI = URI.create(stringMetastoreURI.get());
- }
+ URI hiveMetastoreURI = stringMetastoreURI.isPresent() ? URI.create(stringMetastoreURI.get()) : null;
DatasetDescriptor destinationDataset =
new DatasetDescriptor(DatasetConstants.PLATFORM_ICEBERG, hiveMetastoreURI, destinationTable);
- destinationDataset.addMetadata(DatasetConstants.FS_URI, this.getFs().getUri().toString());
+ destinationDataset.addMetadata(DatasetConstants.FS_URI, this.getSourceFs().getUri().toString());
Review Comment:
Why is the destinationDataset using the source FS? It can be on a different filesystem uri here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972146519
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergTableFileSet.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.Collection;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+
+/**
+ * A {@link IcebergFileSet} that generates {@link CopyEntity}s for a Hive Catalog based Iceberg table
+ */
+public class IcebergTableFileSet extends IcebergFileSet{
Review Comment:
acknowledged and updated the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
phet commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973294122
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
Review Comment:
let's think ahead to delta calculation... this should be all paths that exist in the snapshot. the caller will determine what actually "needs to be copied"
also, don't go so deep on background details in a method's javadoc. maybe just say "Finds all files of the iceberg's current snapshot"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972146208
##########
gobblin-runtime/src/main/resources/templates/icebergDistcp.template:
##########
@@ -0,0 +1,81 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# ====================================================================
+# Job configurations (can be changed)
+# ====================================================================
+
+# General job metadata
+job.group=DistcpIceberg
+job.name=EmbeddedDistcpIcebergTracking
+job.description=Embedded Iceberg-Distcp-ng toy job
+
Review Comment:
I have not verified. This again snuck in unintentionally. I have deleted the file. I am planning to add this as per of my future commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972165125
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntitiesForTableFileSet(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(hadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ Map<Path, FileStatus> getFilePaths() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("Fetching all file paths for the current snapshot of the Iceberg table");
+ List<String> pathsToCopy = icebergSnapshotInfo.getAllPaths();
+
+ for(String pathString : pathsToCopy) {
+ Path path = new Path(pathString);
+ result.put(path, this.fs.getFileStatus(path));
+ }
+ return result;
+ }
+
+ DatasetDescriptor getSourceDataset() {
+ return getDatasetDescriptor(sourceMetastoreURI);
+ }
+
+ DatasetDescriptor getDestinationDataset() {
+ return getDatasetDescriptor(targetMetastoreURI);
+ }
+
+ @NotNull
+ private DatasetDescriptor getDatasetDescriptor(Optional<String> stringMetastoreURI) {
+ String destinationTable = this.getDbName() + "." + this.getInputTableName();
+
+ URI hiveMetastoreURI = null;
+ if (stringMetastoreURI.isPresent()) {
+ hiveMetastoreURI = URI.create(stringMetastoreURI.get());
+ }
+
+ DatasetDescriptor destinationDataset =
+ new DatasetDescriptor(DatasetConstants.PLATFORM_ICEBERG, hiveMetastoreURI, destinationTable);
+ destinationDataset.addMetadata(DatasetConstants.FS_URI, this.getFs().getUri().toString());
Review Comment:
yeah... these things are also unclear for me. Once we test it end to end is when I feel we will get more clarity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972159467
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
+ * The design will support defaults and other catalogs in future releases.
+ */
+ if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
+ throw new IOException("Iceberg database name or Iceberg table name is missing");
+ }
+ this.dbName = properties.getProperty(ICEBERG_DB_NAME);
+ this.tblName = properties.getProperty(ICEBERG_TABLE_NAME);
+
+ Configuration configuration = HadoopUtils.getConfFromProperties(properties);
+
+ IcebergCatalog icebergCatalog = IcebergCatalogFactory.create(configuration);
+ IcebergTable icebergTable = icebergCatalog.openTable(dbName, tblName);
+ // Currently, we only support one dataset per iceberg table
+ matchingDatasets.add(createIcebergDataset(dbName, tblName, icebergTable, properties, fs));
Review Comment:
added the comment!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
phet commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r977062551
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -40,18 +39,20 @@
* and creates a {@link IcebergDataset} for each one.
*/
@Slf4j
-@AllArgsConstructor
public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
public static final String ICEBERG_DATASET_PREFIX = DatasetConstants.PLATFORM_ICEBERG + ".dataset";
public static final String ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
public static final String ICEBERG_DB_NAME = ICEBERG_DATASET_PREFIX + ".database.name";
public static final String ICEBERG_TABLE_NAME = ICEBERG_DATASET_PREFIX + ".table.name";
- private String dbName;
- private String tblName;
private final Properties properties;
- protected final FileSystem fs;
+ protected final FileSystem sourceFs;
+
+ public IcebergDatasetFinder(FileSystem fs, Properties properties) {
Review Comment:
note: equivalent to `@lombok.RequiredArgsConstructor`
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -185,31 +185,28 @@ protected Map<Path, FileStatus> getFilePathsToFileStatus() throws IOException {
for (String pathString : pathsToCopy) {
Path path = new Path(pathString);
- result.put(path, this.fs.getFileStatus(path));
+ result.put(path, this.sourceFs.getFileStatus(path));
}
return result;
}
- DatasetDescriptor getSourceDataset() {
- return getDatasetDescriptor(sourceMetastoreURI);
+ DatasetDescriptor getSourceDataset(FileSystem sourceFs) {
+ return getDatasetDescriptor(sourceMetastoreURI, sourceFs);
}
- DatasetDescriptor getDestinationDataset() {
- return getDatasetDescriptor(targetMetastoreURI);
+ DatasetDescriptor getDestinationDataset(FileSystem targetFs) {
Review Comment:
shouldn't these either be `@VisibleForTesting` or `protected`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973213452
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(hadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePaths() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("Fetching all file paths for the current snapshot of the Iceberg table");
Review Comment:
gotcha makes sense! I have added some context information about the Iceberg table we are trying to copy likewise added the snapshot id and metadata path for the table (since the table uri is not readily available as part of Iceberg table)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973881148
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
Review Comment:
not applicable here for Iceberg table. I have removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r977068518
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -185,31 +185,28 @@ protected Map<Path, FileStatus> getFilePathsToFileStatus() throws IOException {
for (String pathString : pathsToCopy) {
Path path = new Path(pathString);
- result.put(path, this.fs.getFileStatus(path));
+ result.put(path, this.sourceFs.getFileStatus(path));
}
return result;
}
- DatasetDescriptor getSourceDataset() {
- return getDatasetDescriptor(sourceMetastoreURI);
+ DatasetDescriptor getSourceDataset(FileSystem sourceFs) {
+ return getDatasetDescriptor(sourceMetastoreURI, sourceFs);
}
- DatasetDescriptor getDestinationDataset() {
- return getDatasetDescriptor(targetMetastoreURI);
+ DatasetDescriptor getDestinationDataset(FileSystem targetFs) {
Review Comment:
good callout.... made them protected and updated the file!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
phet commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972509783
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
Review Comment:
wasn't there a comment to verify that's now gone missing?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
Review Comment:
strange you say that... because I see it here still when I've selected "all commits"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972531406
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -53,37 +54,39 @@ public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDatase
public List<IcebergDataset> findDatasets() throws IOException {
List<IcebergDataset> matchingDatasets = new ArrayList<>();
/*
- * Both Iceberg database name and table name are mandatory,
- * since we are currently only supporting Hive Catalog based Iceberg tables.
- * The design will support defaults and other catalogs in future releases.
+ * Both Iceberg database name and table name are mandatory based on current implementation.
+ * Later we may explore supporting datasets similar to Hive
*/
- if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
- throw new IOException("Iceberg database name or Iceberg table name is missing");
+ if (StringUtils.isNotBlank(properties.getProperty(ICEBERG_DB_NAME)) || StringUtils.isNotBlank(properties.getProperty(ICEBERG_TABLE_NAME))) {
+ throw new IllegalArgumentException(String.format("Iceberg database name: {%s} or Iceberg table name: {%s} is missing",
+ ICEBERG_DB_NAME, ICEBERG_TABLE_NAME));
}
this.dbName = properties.getProperty(ICEBERG_DB_NAME);
this.tblName = properties.getProperty(ICEBERG_TABLE_NAME);
Configuration configuration = HadoopUtils.getConfFromProperties(properties);
IcebergCatalog icebergCatalog = IcebergCatalogFactory.create(configuration);
- IcebergTable icebergTable = icebergCatalog.openTable(dbName, tblName);
- // Currently, we only support one dataset per iceberg table
- matchingDatasets.add(createIcebergDataset(dbName, tblName, icebergTable, properties, fs));
+ /* Currently, we only support one dataset per iceberg table
+ * Error handling and verification of table existence will be included as part IcebergTable.getCurrentSnapshotInfo() in future releases.
Review Comment:
added a TODO for next iteration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972531749
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
Review Comment:
added that back!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972164106
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
+ * The design will support defaults and other catalogs in future releases.
+ */
+ if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
+ throw new IOException("Iceberg database name or Iceberg table name is missing");
+ }
+ this.dbName = properties.getProperty(ICEBERG_DB_NAME);
+ this.tblName = properties.getProperty(ICEBERG_TABLE_NAME);
+
+ Configuration configuration = HadoopUtils.getConfFromProperties(properties);
+
+ IcebergCatalog icebergCatalog = IcebergCatalogFactory.create(configuration);
+ IcebergTable icebergTable = icebergCatalog.openTable(dbName, tblName);
+ // Currently, we only support one dataset per iceberg table
+ matchingDatasets.add(createIcebergDataset(dbName, tblName, icebergTable, properties, fs));
+ log.info("Found {} matching datasets: {}", matchingDatasets.size(), matchingDatasets);
+
+ return matchingDatasets;
+ }
+
+ @Override
+ public Path commonDatasetRoot() {
+ return null;
+ }
+
+ @Override
+ public Iterator<IcebergDataset> getDatasetsIterator() throws IOException {
+ return findDatasets().iterator();
+ }
+ protected IcebergDataset createIcebergDataset(String dbName, String tblName, IcebergTable icebergTable, Properties properties, FileSystem fs) {
Review Comment:
yeah can be done.... updated the file as per your suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972184976
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntitiesForTableFileSet(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(hadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ Map<Path, FileStatus> getFilePaths() throws IOException {
Review Comment:
It is `@VisibleForTesting`. I have added the tag and also made it `protected`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972185571
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntitiesForTableFileSet(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
Review Comment:
yeah I agree... keeping it simple for this particular PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972179079
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ return getCopyEntities(configuration);
Review Comment:
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972182178
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = "iceberg.dataset";
+ public static final String ICEBERG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = DatasetConstants.PLATFORM_ICEBERG + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = DatasetConstants.PLATFORM_ICEBERG + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
+
+ @Override
+ public List<IcebergDataset> findDatasets() throws IOException {
+ List<IcebergDataset> matchingDatasets = new ArrayList<>();
+ /*
+ * Both Iceberg database name and table name are mandatory,
+ * since we are currently only supporting Hive Catalog based Iceberg tables.
+ * The design will support defaults and other catalogs in future releases.
+ */
+ if (properties.getProperty(ICEBERG_DB_NAME) == null || properties.getProperty(ICEBERG_TABLE_NAME) == null) {
+ throw new IOException("Iceberg database name or Iceberg table name is missing");
+ }
+ this.dbName = properties.getProperty(ICEBERG_DB_NAME);
+ this.tblName = properties.getProperty(ICEBERG_TABLE_NAME);
+
+ Configuration configuration = HadoopUtils.getConfFromProperties(properties);
+
+ IcebergCatalog icebergCatalog = IcebergCatalogFactory.create(configuration);
+ IcebergTable icebergTable = icebergCatalog.openTable(dbName, tblName);
+ // Currently, we only support one dataset per iceberg table
+ matchingDatasets.add(createIcebergDataset(dbName, tblName, icebergTable, properties, fs));
+ log.info("Found {} matching datasets: {}", matchingDatasets.size(), matchingDatasets);
+
+ return matchingDatasets;
+ }
+
+ @Override
+ public Path commonDatasetRoot() {
Review Comment:
as part of overriding all the parent methods... I just kept it as the default implementation. But having a default root path makes more sense here similar to HiveDatasetFinder. I have updated the file accordingly in my latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973212838
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
Review Comment:
gotcha makes sense! I have added some context information about the Iceberg table we are trying to copy likewise added the snapshot id and metadata path for the table (since the table uri is not readily available as part of Iceberg table)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] Will-Lo commented on a diff in pull request #3560: [GOBBLIN-1709] Create Iceberg Datasets Finder, Iceberg Dataset and FileSet to generate Copy Entities to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
Will-Lo commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r974812322
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private final IcebergTable icebergTable;
+ protected final Properties properties;
+ protected final FileSystem fs;
Review Comment:
Is this fs the source or destination fs? We run into this challenge constantly a lot with existing sources, I am presuming since it is used in the dataOutputPath it is the destination fs. If so, please specify explicitly in the variable name, it's very easy to be confused when debugging
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private final IcebergTable icebergTable;
+ protected final Properties properties;
+ protected final FileSystem fs;
+
+ private final Optional<String> sourceMetastoreURI;
+ private final Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> pathToFileStatus = getFilePathsToFileStatus();
+ log.info("{}.{} - found {} candidate source paths", dbName, inputTableName, pathToFileStatus.size());
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(pathToFileStatus, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ log.info("{}.{} - generated {} copy entities", dbName, inputTableName, copyEntities.size());
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file path
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> pathToFileStatus, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for (Map.Entry<Path, FileStatus> entry : pathToFileStatus.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for (SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO: Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files of the Iceberg's current snapshot
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePathsToFileStatus() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("{}.{} - loaded snapshot '{}' from metadata path: '{}'", dbName, inputTableName,
+ icebergSnapshotInfo.getSnapshotId(), icebergSnapshotInfo.getMetadataPath());
+ List<String> pathsToCopy = icebergSnapshotInfo.getAllPaths();
+
+ for (String pathString : pathsToCopy) {
+ Path path = new Path(pathString);
+ result.put(path, this.fs.getFileStatus(path));
+ }
+ return result;
+ }
+
+ DatasetDescriptor getSourceDataset() {
+ return getDatasetDescriptor(sourceMetastoreURI);
+ }
+
+ DatasetDescriptor getDestinationDataset() {
+ return getDatasetDescriptor(targetMetastoreURI);
+ }
+
+ @NotNull
+ private DatasetDescriptor getDatasetDescriptor(Optional<String> stringMetastoreURI) {
+ String destinationTable = this.getDbName() + "." + this.getInputTableName();
+
+ URI hiveMetastoreURI = null;
+ if (stringMetastoreURI.isPresent()) {
+ hiveMetastoreURI = URI.create(stringMetastoreURI.get());
+ }
Review Comment:
Small nit: this can be simplified as a ternary operation
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Properties;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.AllArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.IterableDatasetFinder;
+import org.apache.gobblin.util.HadoopUtils;
+
+/**
+ * Finds {@link IcebergDataset}s. Will look for tables in a database using a {@link IcebergCatalog},
+ * and creates a {@link IcebergDataset} for each one.
+ */
+@Slf4j
+@AllArgsConstructor
+public class IcebergDatasetFinder implements IterableDatasetFinder<IcebergDataset> {
+
+ public static final String ICEBERG_DATASET_PREFIX = DatasetConstants.PLATFORM_ICEBERG + ".dataset";
+ public static final String ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY = ICEBERG_DATASET_PREFIX + ".hive.metastore.uri";
+ public static final String ICEBERG_DB_NAME = ICEBERG_DATASET_PREFIX + ".database.name";
+ public static final String ICEBERG_TABLE_NAME = ICEBERG_DATASET_PREFIX + ".table.name";
+
+ private String dbName;
+ private String tblName;
+ private final Properties properties;
+ protected final FileSystem fs;
Review Comment:
Same comment as above, we should specify which fs this is since its ambiguous if it's source or destination
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973514022
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePaths() throws IOException {
Review Comment:
Renamed it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973514815
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
Review Comment:
fixed the spacing issue with the formatting!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972144986
##########
gobblin-data-management/src/test/java/org/apache/gobblin/runtime/embedded/EmbeddedGobblinDistcpTest.java:
##########
@@ -386,4 +386,4 @@ public Set<Characteristic> applicableCharacteristics() {
}
}
-}
+}
Review Comment:
yeah... it was unintentional. Updated it as per master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972188966
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to copy all the file paths for a mocked iceberg table. This is a full copy overwriting everything on the destination
+ */
+ @Test
+ public void testGenerateCopyEntitiesForTableFileSet() throws IOException, URISyntaxException {
+
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ String test_db_name = "test_db_name";
+ String test_table_name = "test_tbl_name";
+ Set<String> setOfFilePaths = new HashSet<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
+
+ Properties properties = new Properties();
+ properties.setProperty("data.publisher.final.dir", "/test");
+
+ CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
+ .preserve(PreserveAttributes.fromMnemonicString(""))
+ .copyContext(new CopyContext())
+ .build();
+ TableOperations tableOperations = Mockito.mock(TableOperations.class);
+
+ IcebergTable icebergTable = new MockedIcebergTable(tableOperations);
+ IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+
+ FileStatus fileStatus1 = new FileStatus();
+ fileStatus1.setPath(new Path(METADATA_PATH));
+ FileStatus fileStatus2 = new FileStatus();
+ fileStatus2.setPath(new Path(MANIFEST_PATH));
+ FileStatus fileStatus3 = new FileStatus();
+ fileStatus3.setPath(new Path(MANIFEST_FILE_PATH1));
+ FileStatus fileStatus4 = new FileStatus();
+ fileStatus4.setPath(new Path(MANIFEST_FILE_PATH2));
+
+ Path path1 = new Path(METADATA_PATH);
+ Path path2 = new Path(MANIFEST_PATH);
+ Path path3 = new Path(MANIFEST_FILE_PATH1);
+ Path path4 = new Path(MANIFEST_FILE_PATH2);
+
+
+ Mockito.when(fs.makeQualified(any(Path.class))).thenReturn(new Path("/root/iceberg/test/destination/sub_path_destination"));
+ Mockito.when(fs.getFileStatus(path1)).thenReturn(fileStatus1);
+ Mockito.when(fs.getFileStatus(path2)).thenReturn(fileStatus2);
+ Mockito.when(fs.getFileStatus(path3)).thenReturn(fileStatus3);
+ Mockito.when(fs.getFileStatus(path4)).thenReturn(fileStatus4);
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, "/root/iceberg/test/output", null));
Review Comment:
yup... I have a created a helper for `FileSystem` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r972186242
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.TableOperations;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import static org.mockito.Matchers.*;
+
+
+public class IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
Review Comment:
yeah... since we test it anyway in the other test case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973199662
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(hadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePaths() throws IOException {
+ Map<Path, FileStatus> result = Maps.newHashMap();
+ IcebergTable icebergTable = this.getIcebergTable();
+ IcebergSnapshotInfo icebergSnapshotInfo = icebergTable.getCurrentSnapshotInfo();
+
+ log.info("Fetching all file paths for the current snapshot of the Iceberg table");
+ List<String> pathsToCopy = icebergSnapshotInfo.getAllPaths();
+
+ for(String pathString : pathsToCopy) {
+ Path path = new Path(pathString);
+ result.put(path, this.fs.getFileStatus(path));
Review Comment:
Currently, we are scoping it to just specific Iceberg tables only in the test environment wherein we know such cases won't arise for the tables we are trying to copy. We wanted to get this piece done in iterative fashion. Once, we have tested it against some test Iceberg tables, we will extend the logic to incorporate error handling and validation of such corner case scenarios. Does this make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973197476
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+
+ log.info("Fetching copyable file builders from their respective file sets and adding to collection of copy entities");
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration hadoopConfiguration = new Configuration();
Review Comment:
gotcha..... made the change in my latest commit!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973200573
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ public IcebergDataset(String db, String table) {
+ this.dbName = db;
+ this.inputTableName = table;
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ log.info("Fetching all the files to be copied");
Review Comment:
yeah... it is more for checkpointing. I will see if I can add more context to make or else just remove it if its redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] meethngala commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
meethngala commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973492564
##########
gobblin-data-management/src/test/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetTest.java:
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.mockito.Mockito;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import com.google.api.client.util.Maps;
+import com.google.gson.Gson;
+import com.google.gson.JsonObject;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyContext;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.PreserveAttributes;
+
+
+public class
+IcebergDatasetTest {
+
+ static final String METADATA_PATH = "/root/iceberg/test/metadata";
+ static final String MANIFEST_PATH = "/root/iceberg/test/metadata/test_manifest";
+ static final String MANIFEST_LIST_PATH = "/root/iceberg/test/metadata/test_manifest/data";
+ static final String MANIFEST_FILE_PATH1 = "/root/iceberg/test/metadata/test_manifest/data/a";
+ static final String MANIFEST_FILE_PATH2 = "/root/iceberg/test/metadata/test_manifest/data/b";
+
+ @Test
+ public void testGetFilePaths() throws IOException {
+
+ List<String> pathsToCopy = new ArrayList<>();
+ pathsToCopy.add(MANIFEST_FILE_PATH1);
+ pathsToCopy.add(MANIFEST_FILE_PATH2);
+ Map<Path, FileStatus> expected = Maps.newHashMap();
+ expected.put(new Path(MANIFEST_FILE_PATH1), null);
+ expected.put(new Path(MANIFEST_FILE_PATH2), null);
+
+ IcebergTable icebergTable = Mockito.mock(IcebergTable.class);
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ IcebergSnapshotInfo icebergSnapshotInfo = Mockito.mock(IcebergSnapshotInfo.class);
+
+ Mockito.when(icebergTable.getCurrentSnapshotInfo()).thenReturn(icebergSnapshotInfo);
+ Mockito.when(icebergSnapshotInfo.getAllPaths()).thenReturn(pathsToCopy);
+ IcebergDataset icebergDataset = new IcebergDataset("test_db_name", "test_tbl_name", icebergTable, new Properties(), fs);
+
+ Map<Path, FileStatus> actual = icebergDataset.getFilePaths();
+ Assert.assertEquals(actual, expected);
+ }
+
+ /**
+ * Test case to generate copy entities for all the file paths for a mocked iceberg table.
+ * The assumption here is that we create copy entities for all the matching file paths,
+ * without calculating any difference between the source and destination
+ */
+ @Test
+ public void testGenerateCopyEntitiesWhenDestEmpty() throws IOException, URISyntaxException {
+
+ FileSystem fs = Mockito.mock(FileSystem.class);
+ String test_db_name = "test_db_name";
+ String test_table_name = "test_tbl_name";
+ String test_qualified_path = "/root/iceberg/test/destination/sub_path_destination";
+ String test_uri_path = "/root/iceberg/test/output";
+ Properties properties = new Properties();
+ properties.setProperty("data.publisher.final.dir", "/test");
+ List<String> expected = new ArrayList<>(Arrays.asList(METADATA_PATH, MANIFEST_PATH, MANIFEST_LIST_PATH, MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2));
+
+ CopyConfiguration copyConfiguration = CopyConfiguration.builder(null, properties)
+ .preserve(PreserveAttributes.fromMnemonicString(""))
+ .copyContext(new CopyContext())
+ .build();
+
+ List<String> listedManifestFilePaths = Arrays.asList(MANIFEST_FILE_PATH1, MANIFEST_FILE_PATH2);
+ IcebergSnapshotInfo.ManifestFileInfo manifestFileInfo = new IcebergSnapshotInfo.ManifestFileInfo(MANIFEST_LIST_PATH, listedManifestFilePaths);
+ List<IcebergSnapshotInfo.ManifestFileInfo> manifestFiles = Arrays.asList(manifestFileInfo);
+ IcebergTable icebergTable = new MockedIcebergTable(METADATA_PATH, MANIFEST_PATH, manifestFiles);
+ IcebergDataset icebergDataset = new IcebergDataset(test_db_name, test_table_name, icebergTable, new Properties(), fs);
+ MockDestinationFileSystem mockDestinationFileSystem = new MockDestinationFileSystem();
+ mockDestinationFileSystem.addPath(METADATA_PATH);
+ mockDestinationFileSystem.addPath(MANIFEST_PATH);
+ mockDestinationFileSystem.addPath(MANIFEST_LIST_PATH);
+ mockDestinationFileSystem.addPath(MANIFEST_FILE_PATH1);
+ mockDestinationFileSystem.addPath(MANIFEST_FILE_PATH2);
+
+ mockFileSystemMethodCalls(fs, mockDestinationFileSystem.pathToFileStatus, test_qualified_path, test_uri_path);
+
+ Collection<CopyEntity> copyEntities = icebergDataset.generateCopyEntities(copyConfiguration);
+ verifyCopyEntities(copyEntities, expected);
+
+ }
+
+ private void verifyCopyEntities(Collection<CopyEntity> copyEntities, List<String> expected) {
+ List<String> actual = new ArrayList<>();
+ for (CopyEntity copyEntity : copyEntities) {
+ String json = copyEntity.toString();
+ JsonObject jsonObject = new Gson().fromJson(json, JsonObject.class);
+ JsonObject objectData =
+ jsonObject.getAsJsonObject("object-data").getAsJsonObject("origin").getAsJsonObject("object-data");
+ JsonObject pathObject = objectData.getAsJsonObject("path").getAsJsonObject("object-data").getAsJsonObject("uri");
+ String filepath = pathObject.getAsJsonPrimitive("object-data").getAsString();
+ actual.add(filepath);
+ }
+ Assert.assertEquals(actual.size(), expected.size());
+ Assert.assertEqualsNoOrder(actual.toArray(), expected.toArray());
+ }
+
+ private void mockFileSystemMethodCalls(FileSystem fs, Map<Path, FileStatus> pathToFileStatus, String qualifiedPath, String uriPath)
+ throws URISyntaxException, IOException {
+
+ Mockito.when(fs.getUri()).thenReturn(new URI(null, null, uriPath, null));
+ for (Map.Entry<Path, FileStatus> entry : pathToFileStatus.entrySet()) {
+ Path path = entry.getKey();
+ FileStatus fileStatus = entry.getValue();
+ Mockito.when(fs.getFileStatus(path)).thenReturn(fileStatus);
+ Mockito.when(fs.makeQualified(path)).thenReturn(new Path(qualifiedPath));
+ }
+ }
+
+ private static class MockedIcebergTable extends IcebergTable {
+
+ String metadataPath;
+ String manifestListPath;
+ List<IcebergSnapshotInfo.ManifestFileInfo> manifestFiles;
+
+ public MockedIcebergTable(String metadataPath, String manifestListPath, List<IcebergSnapshotInfo.ManifestFileInfo> manifestFiles) {
+ super(null);
+ this.metadataPath = metadataPath;
+ this.manifestListPath = manifestListPath;
+ this.manifestFiles = manifestFiles;
+ }
+
+ @Override
+ public IcebergSnapshotInfo getCurrentSnapshotInfo() {
+ Long snapshotId = 0L;
+ Instant timestamp = Instant.ofEpochMilli(0L);
+ return new IcebergSnapshotInfo(snapshotId, timestamp, metadataPath, manifestListPath, manifestFiles);
+ }
+ }
+
+ private static class MockDestinationFileSystem {
+ Map<Path, FileStatus> pathToFileStatus;
Review Comment:
yeah makes sense.... I have updated it as part of the latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] phet commented on a diff in pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
phet commented on code in PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#discussion_r973292042
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.data.management.copy.iceberg;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.jetbrains.annotations.NotNull;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.collect.Iterators;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import lombok.Data;
+import lombok.Getter;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.data.management.copy.CopyConfiguration;
+import org.apache.gobblin.data.management.copy.CopyEntity;
+import org.apache.gobblin.data.management.copy.CopyableDataset;
+import org.apache.gobblin.data.management.copy.CopyableFile;
+import org.apache.gobblin.data.management.copy.prioritization.PrioritizedCopyableDataset;
+import org.apache.gobblin.data.management.partition.FileSet;
+import org.apache.gobblin.dataset.DatasetConstants;
+import org.apache.gobblin.dataset.DatasetDescriptor;
+import org.apache.gobblin.util.request_allocation.PushDownRequestor;
+
+/**
+ * Iceberg dataset implementing {@link CopyableDataset}.
+ */
+@Slf4j
+@Getter
+public class IcebergDataset implements PrioritizedCopyableDataset {
+ private final String dbName;
+ private final String inputTableName;
+ private IcebergTable icebergTable;
+ protected Properties properties;
+ protected FileSystem fs;
+
+ private Optional<String> sourceMetastoreURI;
+ private Optional<String> targetMetastoreURI;
+
+ /** Target metastore URI */
+ public static final String TARGET_METASTORE_URI_KEY =
+ IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.metastore.uri";
+ /** Target database name */
+ public static final String TARGET_DATABASE_KEY = IcebergDatasetFinder.ICEBERG_DATASET_PREFIX + ".copy.target.database";
+
+ public IcebergDataset(String db, String table, IcebergTable icebergTbl, Properties properties, FileSystem fs) {
+ this.dbName = db;
+ this.inputTableName = table;
+ this.icebergTable = icebergTbl;
+ this.properties = properties;
+ this.fs = fs;
+ this.sourceMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(IcebergDatasetFinder.ICEBERG_HIVE_CATALOG_METASTORE_URI_KEY));
+ this.targetMetastoreURI =
+ Optional.fromNullable(this.properties.getProperty(TARGET_METASTORE_URI_KEY));
+ }
+
+ /**
+ * Represents a source {@link FileStatus} and a {@link Path} destination.
+ */
+ @Data
+ private static class SourceAndDestination {
+ private final FileStatus source;
+ private final Path destination;
+ }
+
+ @Override
+ public String datasetURN() {
+ // TODO: verify!
+ return this.dbName + "." + this.inputTableName;
+ }
+
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration) {
+ return getCopyEntities(configuration);
+ }
+ /**
+ * Finds all files read by the table and generates CopyableFiles.
+ * For the specific semantics see {@link #getCopyEntities}.
+ */
+ @Override
+ public Iterator<FileSet<CopyEntity>> getFileSetIterator(FileSystem targetFs, CopyConfiguration configuration,
+ Comparator<FileSet<CopyEntity>> prioritizer, PushDownRequestor<FileSet<CopyEntity>> requestor) {
+ // TODO: Implement PushDownRequestor and priority based copy entity iteration
+ return getCopyEntities(configuration);
+ }
+
+ /**
+ * Finds all files read by the table and generates {@link CopyEntity}s for duplicating the table.
+ */
+ Iterator<FileSet<CopyEntity>> getCopyEntities(CopyConfiguration configuration) {
+ FileSet<CopyEntity> fileSet = new IcebergTableFileSet(this.getInputTableName(), this, configuration);
+ return Iterators.singletonIterator(fileSet); }
+
+ /**
+ * Finds all files read by the table file set and generates {@link CopyEntity}s for duplicating the table.
+ */
+ @VisibleForTesting
+ Collection<CopyEntity> generateCopyEntities(CopyConfiguration configuration) throws IOException {
+ String fileSet = this.getInputTableName();
+ List<CopyEntity> copyEntities = Lists.newArrayList();
+ Map<Path, FileStatus> mapOfPathsToCopy = getFilePaths();
+ log.info("Found {} to be copied over to the destination", mapOfPathsToCopy.size());
+
+
+ for (CopyableFile.Builder builder : getCopyableFilesFromPaths(mapOfPathsToCopy, configuration)) {
+ CopyableFile fileEntity =
+ builder.fileSet(fileSet).datasetOutputPath(this.fs.getUri().getPath()).build();
+ fileEntity.setSourceData(getSourceDataset());
+ fileEntity.setDestinationData(getDestinationDataset());
+ log.info("Adding file path: {} to the collection of copy entities and copying to destination: {}",
+ fileEntity.getFileStatus().getPath(), fileEntity.getDestination().toString());
+ copyEntities.add(fileEntity);
+ }
+ return copyEntities;
+ }
+
+ /**
+ * Get builders for a {@link CopyableFile} for each file referred to by a {@link org.apache.hadoop.hive.metastore.api.StorageDescriptor}.
+ */
+ protected List<CopyableFile.Builder> getCopyableFilesFromPaths(Map<Path, FileStatus> paths, CopyConfiguration configuration) throws IOException {
+
+ List<CopyableFile.Builder> builders = Lists.newArrayList();
+ List<SourceAndDestination> dataFiles = Lists.newArrayList();
+ Configuration defaultHadoopConfiguration = new Configuration();
+ FileSystem actualSourceFs;
+
+ for(Map.Entry<Path, FileStatus> entry : paths.entrySet()) {
+ dataFiles.add(new SourceAndDestination(entry.getValue(), this.fs.makeQualified(entry.getKey())));
+ }
+
+ for(SourceAndDestination sourceAndDestination : dataFiles) {
+ actualSourceFs = sourceAndDestination.getSource().getPath().getFileSystem(defaultHadoopConfiguration);
+
+ // TODO Add ancestor owner and permissions in future releases
+ builders.add(CopyableFile.fromOriginAndDestination(actualSourceFs, sourceAndDestination.getSource(),
+ sourceAndDestination.getDestination(), configuration));
+ }
+ return builders;
+ }
+ /**
+ * Finds all files read by the Iceberg table including metadata json file, manifest files, nested manifest file paths and actual data files.
+ * Returns a map of path, file status for each file that needs to be copied
+ */
+ protected Map<Path, FileStatus> getFilePaths() throws IOException {
Review Comment:
nit: `getFilePathsToFileStatus`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [gobblin] codecov-commenter commented on pull request #3560: [GOBBLIN-1709] Create workunits for Hive Catalog based Iceberg datasets to support Distcp for Iceberg
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3560:
URL: https://github.com/apache/gobblin/pull/3560#issuecomment-1245746229
# [Codecov](https://codecov.io/gh/apache/gobblin/pull/3560?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3560](https://codecov.io/gh/apache/gobblin/pull/3560?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (86d229e) into [master](https://codecov.io/gh/apache/gobblin/commit/d0e530a78ae0417ea6e3ab6aeaeb4e2f260e227f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d0e530a) will **decrease** coverage by `2.96%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #3560 +/- ##
============================================
- Coverage 46.81% 43.84% -2.97%
+ Complexity 10514 2055 -8459
============================================
Files 2095 406 -1689
Lines 81945 17528 -64417
Branches 9129 2142 -6987
============================================
- Hits 38361 7686 -30675
+ Misses 40040 8986 -31054
+ Partials 3544 856 -2688
```
| [Impacted Files](https://codecov.io/gh/apache/gobblin/pull/3560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh) | `77.41% <0.00%> (-6.46%)` | :arrow_down: |
| [...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==) | `29.54% <0.00%> (-0.33%)` | :arrow_down: |
| [...a/org/apache/gobblin/dataset/DatasetConstants.java](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YXNldC9EYXRhc2V0Q29uc3RhbnRzLmphdmE=) | | |
| [...service/modules/dataset/HttpDatasetDescriptor.java](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9kYXRhc2V0L0h0dHBEYXRhc2V0RGVzY3JpcHRvci5qYXZh) | | |
| [...agement/copy/SubsetFilesCopyableDatasetFinder.java](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvU3Vic2V0RmlsZXNDb3B5YWJsZURhdGFzZXRGaW5kZXIuamF2YQ==) | | |
| [.../gobblin/source/DatePartitionedAvroFileSource.java](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9EYXRlUGFydGl0aW9uZWRBdnJvRmlsZVNvdXJjZS5qYXZh) | | |
| [...n/google/webmaster/GoogleWebMasterSourceDaily.java](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvb2dsZS1pbmdlc3Rpb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vaW5nZXN0aW9uL2dvb2dsZS93ZWJtYXN0ZXIvR29vZ2xlV2ViTWFzdGVyU291cmNlRGFpbHkuamF2YQ==) | | |
| [...ava/org/apache/gobblin/broker/SimpleScopeType.java](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vYnJva2VyL1NpbXBsZVNjb3BlVHlwZS5qYXZh) | | |
| [...a/org/apache/gobblin/crypto/EncryptionFactory.java](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY3J5cHRvL0VuY3J5cHRpb25GYWN0b3J5LmphdmE=) | | |
| [...main/java/org/apache/gobblin/r2/D2ClientProxy.java](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taHR0cC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9yMi9EMkNsaWVudFByb3h5LmphdmE=) | | |
| ... and [1681 more](https://codecov.io/gh/apache/gobblin/pull/3560/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org