You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/08/14 00:18:52 UTC
[GitHub] [hudi] satishkotha opened a new pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
satishkotha opened a new pull request #1964:
URL: https://github.com/apache/hudi/pull/1964
## What is the purpose of the pull request
Add IncrementalMetaClient as separate class to query partitions affected in a specified time window
## Brief change log
- Add IncrementalMetaClient as separate abstraction
- Modify HiveSync to use this new MetaClient
- We also need this for other use cases to easily query affected partitions (example: sync a table between multiple regions)
## Verify this pull request
This change added tests
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475901534
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java
##########
@@ -232,6 +233,12 @@
*/
Option<byte[]> getInstantDetails(HoodieInstant instant);
+ /**
+ * Returns partitions that have been modified in the timeline. This includes internal operations such as clean.
+ * Note that this only returns data for completed instants.
+ */
+ List<String> getPartitionsMutated();
Review comment:
@n3nash I want this to be in hudi-common, so this can be reused in hadoop-mr and hive-sync. Do you want me to create TimelineUtils in common?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475931147
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.exception.HoodieIOException;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * TimelineUtils provides a common way to query incremental meta-data changes for a hoodie table.
+ *
+ * This is useful in multiple places including:
+ * 1) HiveSync - this can be used to query partitions that changed since previous sync.
+ * 2) Incremental reads - InputFormats can use this API to queryxw
Review comment:
typo : queryxw -> query
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.exception.HoodieIOException;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * TimelineUtils provides a common way to query incremental meta-data changes for a hoodie table.
+ *
+ * This is useful in multiple places including:
+ * 1) HiveSync - this can be used to query partitions that changed since previous sync.
+ * 2) Incremental reads - InputFormats can use this API to queryxw
+ */
+public class TimelineUtils {
+
+ /**
+ * Returns partitions that have new data strictly after commitTime.
+ * Does not include internal operations such as clean in the timeline.
+ */
+ public static List<String> getPartitionsWritten(HoodieTimeline timeline) {
+ HoodieTimeline timelineToSync = timeline.getCommitsAndCompactionTimeline();
+ return getPartitionsMutated(timelineToSync);
+ }
+
+ /**
+ * Returns partitions that have been modified including internal operations such as clean in the passed timeline.
+ */
+ public static List<String> getPartitionsMutated(HoodieTimeline timeline) {
Review comment:
I am assuming you are planning to use this API internally. Right ?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.exception.HoodieIOException;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * TimelineUtils provides a common way to query incremental meta-data changes for a hoodie table.
+ *
+ * This is useful in multiple places including:
+ * 1) HiveSync - this can be used to query partitions that changed since previous sync.
+ * 2) Incremental reads - InputFormats can use this API to queryxw
+ */
+public class TimelineUtils {
+
+ /**
+ * Returns partitions that have new data strictly after commitTime.
+ * Does not include internal operations such as clean in the timeline.
+ */
+ public static List<String> getPartitionsWritten(HoodieTimeline timeline) {
+ HoodieTimeline timelineToSync = timeline.getCommitsAndCompactionTimeline();
+ return getPartitionsMutated(timelineToSync);
+ }
+
+ /**
+ * Returns partitions that have been modified including internal operations such as clean in the passed timeline.
+ */
+ public static List<String> getPartitionsMutated(HoodieTimeline timeline) {
Review comment:
getPartitionsMutated -> getAffectedPartitions ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475962559
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.exception.HoodieIOException;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * TimelineUtils provides a common way to query incremental meta-data changes for a hoodie table.
+ *
+ * This is useful in multiple places including:
+ * 1) HiveSync - this can be used to query partitions that changed since previous sync.
+ * 2) Incremental reads - InputFormats can use this API to queryxw
+ */
+public class TimelineUtils {
+
+ /**
+ * Returns partitions that have new data strictly after commitTime.
+ * Does not include internal operations such as clean in the timeline.
+ */
+ public static List<String> getPartitionsWritten(HoodieTimeline timeline) {
+ HoodieTimeline timelineToSync = timeline.getCommitsAndCompactionTimeline();
+ return getPartitionsMutated(timelineToSync);
+ }
+
+ /**
+ * Returns partitions that have been modified including internal operations such as clean in the passed timeline.
+ */
+ public static List<String> getPartitionsMutated(HoodieTimeline timeline) {
+ return timeline.filterCompletedInstants().getInstants().flatMap(s -> {
+ switch (s.getAction()) {
+ case HoodieTimeline.COMMIT_ACTION:
+ case HoodieTimeline.DELTA_COMMIT_ACTION:
+ try {
+ HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(timeline.getInstantDetails(s).get(), HoodieCommitMetadata.class);
+ return commitMetadata.getPartitionToWriteStats().keySet().stream();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to get partitions written between " + timeline.firstInstant() + " " + timeline.lastInstant(), e);
+ }
+ case HoodieTimeline.CLEAN_ACTION:
+ try {
+ HoodieCleanMetadata cleanMetadata = TimelineMetadataUtils.deserializeHoodieCleanMetadata(timeline.getInstantDetails(s).get());
+ return cleanMetadata.getPartitionMetadata().keySet().stream();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to get partitions cleaned between " + timeline.firstInstant() + " " + timeline.lastInstant(), e);
+ }
+ case HoodieTimeline.COMPACTION_ACTION:
+ // compaction is not a completed instant. So no need to consider this action.
+ case HoodieTimeline.SAVEPOINT_ACTION:
+ case HoodieTimeline.ROLLBACK_ACTION:
+ case HoodieTimeline.RESTORE_ACTION:
+ return Stream.empty();
Review comment:
compaction is not treated as completed instants. So this can be ignored. I implemented all other actions and added tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r476026846
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.exception.HoodieIOException;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * TimelineUtils provides a common way to query incremental meta-data changes for a hoodie table.
+ *
+ * This is useful in multiple places including:
+ * 1) HiveSync - this can be used to query partitions that changed since previous sync.
+ * 2) Incremental reads - InputFormats can use this API to queryxw
+ */
+public class TimelineUtils {
+
+ /**
+ * Returns partitions that have new data strictly after commitTime.
+ * Does not include internal operations such as clean in the timeline.
+ */
+ public static List<String> getPartitionsWritten(HoodieTimeline timeline) {
+ HoodieTimeline timelineToSync = timeline.getCommitsAndCompactionTimeline();
+ return getPartitionsMutated(timelineToSync);
+ }
+
+ /**
+ * Returns partitions that have been modified including internal operations such as clean in the passed timeline.
+ */
+ public static List<String> getPartitionsMutated(HoodieTimeline timeline) {
Review comment:
Sure, getAffectedPartitions is fine.. @satishkotha
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash merged pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
n3nash merged pull request #1964:
URL: https://github.com/apache/hudi/pull/1964
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#issuecomment-674193349
@bvaradar can you review this ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475920279
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java
##########
@@ -296,6 +300,42 @@ public boolean isBeforeTimelineStarts(String instant) {
return details.apply(instant);
}
+ /**
+ * Returns partitions that have been modified in the timeline. This includes internal operations such as clean.
+ * Note that this only returns data for completed instants.
+ */
+ public List<String> getPartitionsMutated() {
+ return filterCompletedInstants().getInstants().flatMap(s -> {
+ switch (s.getAction()) {
+ case HoodieTimeline.COMMIT_ACTION:
+ case HoodieTimeline.DELTA_COMMIT_ACTION:
+ try {
+ HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(getInstantDetails(s).get(), HoodieCommitMetadata.class);
+ return commitMetadata.getPartitionToWriteStats().keySet().stream();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to get partitions written between " + firstInstant() + " " + lastInstant(), e);
+ }
+ case HoodieTimeline.CLEAN_ACTION:
Review comment:
The method takes in a timeline. So hive sync only passes "commit" timeline to this method and gets only partitions modified by commit instants. I added another method just for clarity that only looks at commits. Let me know if you have any suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475919445
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java
##########
@@ -296,6 +300,42 @@ public boolean isBeforeTimelineStarts(String instant) {
return details.apply(instant);
}
+ /**
Review comment:
Thank you. Moved it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475892688
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java
##########
@@ -232,6 +233,12 @@
*/
Option<byte[]> getInstantDetails(HoodieInstant instant);
+ /**
+ * Returns partitions that have been modified in the timeline. This includes internal operations such as clean.
+ * Note that this only returns data for completed instants.
+ */
+ List<String> getPartitionsMutated();
Review comment:
I don't think this makes sense in the timeline as of now. If you take a look at the timeline API's, they only talk about the metadata that has changed. `getPartitionsMutated` conceptually is providing what has changed in the underlying data as opposed to what has changed in the timeline per se. Generally, all of this information should come from the timeline but that requires a full redesign on the timeline. Should we add this API here -> https://github.com/apache/hudi/blob/master/hudi-client/src/main/java/org/apache/hudi/client/HoodieReadClient.java#L195 ? And you can wrap this functionality in a TimelineUtils ?
When we have clearer design on timeline, we can merge back the TimelineUtils to the real timeline...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475961204
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.exception.HoodieIOException;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * TimelineUtils provides a common way to query incremental meta-data changes for a hoodie table.
+ *
+ * This is useful in multiple places including:
+ * 1) HiveSync - this can be used to query partitions that changed since previous sync.
+ * 2) Incremental reads - InputFormats can use this API to queryxw
+ */
+public class TimelineUtils {
+
+ /**
+ * Returns partitions that have new data strictly after commitTime.
+ * Does not include internal operations such as clean in the timeline.
+ */
+ public static List<String> getPartitionsWritten(HoodieTimeline timeline) {
+ HoodieTimeline timelineToSync = timeline.getCommitsAndCompactionTimeline();
+ return getPartitionsMutated(timelineToSync);
+ }
+
+ /**
+ * Returns partitions that have been modified including internal operations such as clean in the passed timeline.
+ */
+ public static List<String> getPartitionsMutated(HoodieTimeline timeline) {
Review comment:
Yes, this is for internal use. I have no strong opinion on name. I think it was initially getAffectedPartitions, changed to getPartitionsMutated because of @n3nash suggestion. i'm fine with going back if he agrees
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.exception.HoodieIOException;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * TimelineUtils provides a common way to query incremental meta-data changes for a hoodie table.
+ *
+ * This is useful in multiple places including:
+ * 1) HiveSync - this can be used to query partitions that changed since previous sync.
+ * 2) Incremental reads - InputFormats can use this API to queryxw
Review comment:
thank you! fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r476026846
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.exception.HoodieIOException;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * TimelineUtils provides a common way to query incremental meta-data changes for a hoodie table.
+ *
+ * This is useful in multiple places including:
+ * 1) HiveSync - this can be used to query partitions that changed since previous sync.
+ * 2) Incremental reads - InputFormats can use this API to queryxw
+ */
+public class TimelineUtils {
+
+ /**
+ * Returns partitions that have new data strictly after commitTime.
+ * Does not include internal operations such as clean in the timeline.
+ */
+ public static List<String> getPartitionsWritten(HoodieTimeline timeline) {
+ HoodieTimeline timelineToSync = timeline.getCommitsAndCompactionTimeline();
+ return getPartitionsMutated(timelineToSync);
+ }
+
+ /**
+ * Returns partitions that have been modified including internal operations such as clean in the passed timeline.
+ */
+ public static List<String> getPartitionsMutated(HoodieTimeline timeline) {
Review comment:
Sure, getAffectedPartitions is fine @satishkotha, I think initially it was getWrittenPartitions or something..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475925314
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.timeline;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.exception.HoodieIOException;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * TimelineUtils provides a common way to query incremental meta-data changes for a hoodie table.
+ *
+ * This is useful in multiple places including:
+ * 1) HiveSync - this can be used to query partitions that changed since previous sync.
+ * 2) Incremental reads - InputFormats can use this API to queryxw
+ */
+public class TimelineUtils {
+
+ /**
+ * Returns partitions that have new data strictly after commitTime.
+ * Does not include internal operations such as clean in the timeline.
+ */
+ public static List<String> getPartitionsWritten(HoodieTimeline timeline) {
+ HoodieTimeline timelineToSync = timeline.getCommitsAndCompactionTimeline();
+ return getPartitionsMutated(timelineToSync);
+ }
+
+ /**
+ * Returns partitions that have been modified including internal operations such as clean in the passed timeline.
+ */
+ public static List<String> getPartitionsMutated(HoodieTimeline timeline) {
+ return timeline.filterCompletedInstants().getInstants().flatMap(s -> {
+ switch (s.getAction()) {
+ case HoodieTimeline.COMMIT_ACTION:
+ case HoodieTimeline.DELTA_COMMIT_ACTION:
+ try {
+ HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(timeline.getInstantDetails(s).get(), HoodieCommitMetadata.class);
+ return commitMetadata.getPartitionToWriteStats().keySet().stream();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to get partitions written between " + timeline.firstInstant() + " " + timeline.lastInstant(), e);
+ }
+ case HoodieTimeline.CLEAN_ACTION:
+ try {
+ HoodieCleanMetadata cleanMetadata = TimelineMetadataUtils.deserializeHoodieCleanMetadata(timeline.getInstantDetails(s).get());
+ return cleanMetadata.getPartitionMetadata().keySet().stream();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to get partitions cleaned between " + timeline.firstInstant() + " " + timeline.lastInstant(), e);
+ }
+ case HoodieTimeline.COMPACTION_ACTION:
+ // compaction is not a completed instant. So no need to consider this action.
+ case HoodieTimeline.SAVEPOINT_ACTION:
+ case HoodieTimeline.ROLLBACK_ACTION:
+ case HoodieTimeline.RESTORE_ACTION:
+ return Stream.empty();
Review comment:
Do you want to throw an exception here for now so it's not treated incorrectly ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
bvaradar commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475895369
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java
##########
@@ -296,6 +300,42 @@ public boolean isBeforeTimelineStarts(String instant) {
return details.apply(instant);
}
+ /**
Review comment:
Timeline APIs are only about instants in general. I think adding partitions here is breaking that abstraction. Can you move this to some helper class
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java
##########
@@ -232,6 +233,12 @@
*/
Option<byte[]> getInstantDetails(HoodieInstant instant);
+ /**
+ * Returns partitions that have been modified in the timeline. This includes internal operations such as clean.
+ * Note that this only returns data for completed instants.
+ */
+ List<String> getPartitionsMutated();
Review comment:
+1 on abstraction point. I think having a separate helper class would be better.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java
##########
@@ -296,6 +300,42 @@ public boolean isBeforeTimelineStarts(String instant) {
return details.apply(instant);
}
+ /**
+ * Returns partitions that have been modified in the timeline. This includes internal operations such as clean.
+ * Note that this only returns data for completed instants.
+ */
+ public List<String> getPartitionsMutated() {
+ return filterCompletedInstants().getInstants().flatMap(s -> {
+ switch (s.getAction()) {
+ case HoodieTimeline.COMMIT_ACTION:
+ case HoodieTimeline.DELTA_COMMIT_ACTION:
+ try {
+ HoodieCommitMetadata commitMetadata = HoodieCommitMetadata.fromBytes(getInstantDetails(s).get(), HoodieCommitMetadata.class);
+ return commitMetadata.getPartitionToWriteStats().keySet().stream();
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to get partitions written between " + firstInstant() + " " + lastInstant(), e);
+ }
+ case HoodieTimeline.CLEAN_ACTION:
Review comment:
We dont need to look at clean and (event compaction) for figuring out changed partitions for the case of hive syncing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#issuecomment-679298263
Could we structure this as a "BaseTableMetaClient", "HoodieTableMetaClient" and "HoodieTableIncrementalMetaClient" ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] satishkotha commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
satishkotha commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475919254
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieTimeline.java
##########
@@ -232,6 +233,12 @@
*/
Option<byte[]> getInstantDetails(HoodieInstant instant);
+ /**
+ * Returns partitions that have been modified in the timeline. This includes internal operations such as clean.
+ * Note that this only returns data for completed instants.
+ */
+ List<String> getPartitionsMutated();
Review comment:
created TimelineUtils
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on a change in pull request #1964: [HUDI-1191] Add incremental meta client API to query partitions changed
Posted by GitBox <gi...@apache.org>.
n3nash commented on a change in pull request #1964:
URL: https://github.com/apache/hudi/pull/1964#discussion_r475925115
##########
File path: hudi-common/src/test/java/org/apache/hudi/common/table/TestTimelineUtils.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table;
+
+import org.apache.hudi.avro.model.HoodieCleanMetadata;
+import org.apache.hudi.avro.model.HoodieCleanPartitionMetadata;
+import org.apache.hudi.common.model.HoodieCleaningPolicy;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.table.timeline.TimelineMetadataUtils;
+import org.apache.hudi.common.table.timeline.TimelineUtils;
+import org.apache.hudi.common.testutils.HoodieCommonTestHarness;
+import org.apache.hudi.common.util.Option;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class TestTimelineUtils extends HoodieCommonTestHarness {
+
+ @BeforeEach
+ public void setUp() throws Exception {
+ initMetaClient();
+ }
+
+ @Test
+ public void testGetPartitions() throws IOException {
+ HoodieActiveTimeline activeTimeline = metaClient.getActiveTimeline();
+ HoodieTimeline activeCommitTimeline = activeTimeline.getCommitTimeline();
+ assertTrue(activeCommitTimeline.empty());
+
+ String olderPartition = "0"; // older partitions that is modified by all cleans
+ for (int i = 1; i <= 5; i++) {
+ String ts = i + "";
+ HoodieInstant instant = new HoodieInstant(true, HoodieTimeline.COMMIT_ACTION, ts);
+ activeTimeline.createNewInstant(instant);
+ activeTimeline.saveAsComplete(instant, Option.of(getCommitMeta(basePath, ts, ts, 2)));
+
+ HoodieInstant cleanInstant = new HoodieInstant(true, HoodieTimeline.CLEAN_ACTION, ts);
+ activeTimeline.createNewInstant(cleanInstant);
+ activeTimeline.saveAsComplete(cleanInstant, getCleanMeta(olderPartition, ts));
+ }
+
+ metaClient.reloadActiveTimeline();
+
+ // verify modified partitions included cleaned data
+ List<String> partitions = TimelineUtils.getPartitionsMutated(metaClient.getActiveTimeline().findInstantsAfter("1", 10));
+ assertEquals(5, partitions.size());
+ assertEquals(partitions, Arrays.asList(new String[]{"0", "2", "3", "4", "5"}));
+
+ partitions = TimelineUtils.getPartitionsMutated(metaClient.getActiveTimeline().findInstantsInRange("1", "4"));
+ assertEquals(4, partitions.size());
+ assertEquals(partitions, Arrays.asList(new String[]{"0", "2", "3", "4"}));
+
+ // verify only commit actions
+ partitions = TimelineUtils.getPartitionsWritten(metaClient.getActiveTimeline().findInstantsAfter("1", 10));
+ assertEquals(4, partitions.size());
+ assertEquals(partitions, Arrays.asList(new String[]{"2", "3", "4", "5"}));
+
+ partitions = TimelineUtils.getPartitionsWritten(metaClient.getActiveTimeline().findInstantsInRange("1", "4"));
+ assertEquals(3, partitions.size());
+ assertEquals(partitions, Arrays.asList(new String[]{"2", "3", "4"}));
+ }
+
+ @Test
+ public void testGetPartitionsUnpartitioned() throws IOException {
+ HoodieActiveTimeline activeTimeline = metaClient.getActiveTimeline();
+ HoodieTimeline activeCommitTimeline = activeTimeline.getCommitTimeline();
+ assertTrue(activeCommitTimeline.empty());
+
+ String partitionPath = "";
+ for (int i = 1; i <= 5; i++) {
+ String ts = i + "";
+ HoodieInstant instant = new HoodieInstant(true, HoodieTimeline.COMMIT_ACTION, ts);
+ activeTimeline.createNewInstant(instant);
+ activeTimeline.saveAsComplete(instant, Option.of(getCommitMeta(basePath, partitionPath, ts, 2)));
+
+ HoodieInstant cleanInstant = new HoodieInstant(true, HoodieTimeline.CLEAN_ACTION, ts);
+ activeTimeline.createNewInstant(cleanInstant);
+ activeTimeline.saveAsComplete(cleanInstant, getCleanMeta(partitionPath, ts));
+ }
+
+ metaClient.reloadActiveTimeline();
+
+ // verify modified partitions included cleaned data
+ List<String> partitions = TimelineUtils.getPartitionsMutated(metaClient.getActiveTimeline().findInstantsAfter("1", 10));
+ assertTrue(partitions.isEmpty());
+
+ partitions = TimelineUtils.getPartitionsMutated(metaClient.getActiveTimeline().findInstantsInRange("1", "4"));
+ assertTrue(partitions.isEmpty());
+ }
+
+ private byte[] getCommitMeta(String basePath, String partition, String commitTs, int count)
Review comment:
Let's keep the same name as before `getCommitMetadata` & `getCleanMetadata` unless there is a specific need for keeping the names short..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org