You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2023/01/11 00:55:00 UTC
[GitHub] [hudi] the-other-tim-brown opened a new pull request, #7640: Add in support for a keyless workflow by building an ID based off of …
the-other-tim-brown opened a new pull request, #7640:
URL: https://github.com/apache/hudi/pull/7640
…values within the record
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on a diff in pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on code in PR #7640:
URL: https://github.com/apache/hudi/pull/7640#discussion_r1072575334
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeylessKeyGenerator.java:
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayDeque;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+/**
+ * This class is used to compute a deterministic key for a record based on the contents of the field. Unlike the other KeyGenerators in Hudi, this class does not take in any field names as args to
+ * create a "keyless" experience for insert only workloads. The keys are guaranteed to be deterministic but not unique, so they can only be used for insert workflows with deduplication disabled.
+ * The class attempts to get sufficient uniqueness for keys to prevent frequent collisions by choosing the fields it uses in order of decreasing likelihood for uniqueness. The ordering is:
+ * <ul>
+ * <li>timestamp</li>
+ * <li>numeric values</li>
+ * <li>string, byte arrays, other types not mentioned</li>
+ * <li>date, lists, maps, booleans</li>
+ * </ul>
+ * The number of fields is capped to created predictable performance and the generator only uses non-null values to help increase uniqueness for sparse datasets.
+ */
+public class KeylessKeyGenerator extends CustomAvroKeyGenerator {
+ private static final String HOODIE_PREFIX = "_hoodie";
+ private static final String DOT = ".";
+ private final int maxFieldsToConsider;
+ private final int numFieldsForKey;
+ private final Set<String> partitionFieldNames;
+ private int[][] fieldOrdering;
+
+ public KeylessKeyGenerator(TypedProperties props) {
+ super(props);
+ this.numFieldsForKey = props.getInteger(KeyGeneratorOptions.NUM_FIELDS_IN_KEYLESS_GENERATOR.key(), KeyGeneratorOptions.NUM_FIELDS_IN_KEYLESS_GENERATOR.defaultValue());
+ // cap the number of fields to order in case of large schemas
+ this.maxFieldsToConsider = numFieldsForKey * 3;
+ this.partitionFieldNames = this.getPartitionPathFields().stream().map(field -> field.split(SPLIT_REGEX)[0]).collect(Collectors.toSet());
+ }
+
+ @Override
+ public String getRecordKey(GenericRecord record) {
+ return buildKey(getFieldOrdering(record), record);
+ }
+
+ int[][] getFieldOrdering(GenericRecord genericRecord) {
+ if (fieldOrdering == null) {
+ fieldOrdering = buildFieldOrdering(genericRecord.getSchema().getFields());
+ }
+ return fieldOrdering;
+ }
+
+ /**
+ * Deterministically builds a key for the input value based on the provided fieldOrdering. The first {@link #numFieldsForKey} non-null values will be used to generate a string that is passed to
+ * {@link UUID#nameUUIDFromBytes(byte[])}.
+ * @param fieldOrdering an array of integer arrays. The integer arrays represent paths to a single field within the input object.
+ * @param input the input object that needs a key
+ * @return a deterministically generated {@link UUID}
+ * @param <T> the input object type
+ */
+ private <T> String buildKey(int[][] fieldOrdering, GenericRecord input) {
+ StringBuilder key = new StringBuilder();
+ int nonNullFields = 0;
+ for (int[] index : fieldOrdering) {
+ Object value = getFieldForRecord(input, index);
+ if (value == null) {
+ continue;
+ }
+ nonNullFields++;
+ key.append(value.hashCode());
Review Comment:
Do you mean we should append some sort of delimiter after each hashcode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #7640:
URL: https://github.com/apache/hudi/pull/7640#discussion_r1072550590
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeylessKeyGenerator.java:
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayDeque;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+/**
+ * This class is used to compute a deterministic key for a record based on the contents of the field. Unlike the other KeyGenerators in Hudi, this class does not take in any field names as args to
+ * create a "keyless" experience for insert only workloads. The keys are guaranteed to be deterministic but not unique, so they can only be used for insert workflows with deduplication disabled.
+ * The class attempts to get sufficient uniqueness for keys to prevent frequent collisions by choosing the fields it uses in order of decreasing likelihood for uniqueness. The ordering is:
+ * <ul>
+ * <li>timestamp</li>
+ * <li>numeric values</li>
+ * <li>string, byte arrays, other types not mentioned</li>
+ * <li>date, lists, maps, booleans</li>
+ * </ul>
+ * The number of fields is capped to created predictable performance and the generator only uses non-null values to help increase uniqueness for sparse datasets.
+ */
+public class KeylessKeyGenerator extends CustomAvroKeyGenerator {
+ private static final String HOODIE_PREFIX = "_hoodie";
+ private static final String DOT = ".";
+ private final int maxFieldsToConsider;
+ private final int numFieldsForKey;
+ private final Set<String> partitionFieldNames;
+ private int[][] fieldOrdering;
+
+ public KeylessKeyGenerator(TypedProperties props) {
+ super(props);
+ this.numFieldsForKey = props.getInteger(KeyGeneratorOptions.NUM_FIELDS_IN_KEYLESS_GENERATOR.key(), KeyGeneratorOptions.NUM_FIELDS_IN_KEYLESS_GENERATOR.defaultValue());
+ // cap the number of fields to order in case of large schemas
+ this.maxFieldsToConsider = numFieldsForKey * 3;
+ this.partitionFieldNames = this.getPartitionPathFields().stream().map(field -> field.split(SPLIT_REGEX)[0]).collect(Collectors.toSet());
+ }
+
+ @Override
+ public String getRecordKey(GenericRecord record) {
+ return buildKey(getFieldOrdering(record), record);
+ }
+
+ int[][] getFieldOrdering(GenericRecord genericRecord) {
+ if (fieldOrdering == null) {
+ fieldOrdering = buildFieldOrdering(genericRecord.getSchema().getFields());
+ }
+ return fieldOrdering;
+ }
+
+ /**
+ * Deterministically builds a key for the input value based on the provided fieldOrdering. The first {@link #numFieldsForKey} non-null values will be used to generate a string that is passed to
+ * {@link UUID#nameUUIDFromBytes(byte[])}.
+ * @param fieldOrdering an array of integer arrays. The integer arrays represent paths to a single field within the input object.
+ * @param input the input object that needs a key
+ * @return a deterministically generated {@link UUID}
+ * @param <T> the input object type
+ */
+ private <T> String buildKey(int[][] fieldOrdering, GenericRecord input) {
+ StringBuilder key = new StringBuilder();
+ int nonNullFields = 0;
+ for (int[] index : fieldOrdering) {
+ Object value = getFieldForRecord(input, index);
+ if (value == null) {
+ continue;
+ }
+ nonNullFields++;
+ key.append(value.hashCode());
Review Comment:
@the-other-tim-brown @the-other-tim-brown this is incorrect way of hash/key generation, we can't distinguish b/w cases of hash_1=12 and hash_1=1, hash_2=2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #7640:
URL: https://github.com/apache/hudi/pull/7640#discussion_r1072995895
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/keygen/KeylessKeyGenerator.java:
##########
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.keygen;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.keygen.constant.KeyGeneratorOptions;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayDeque;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.List;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.Set;
+import java.util.UUID;
+import java.util.stream.Collectors;
+
+/**
+ * This class is used to compute a deterministic key for a record based on the contents of the field. Unlike the other KeyGenerators in Hudi, this class does not take in any field names as args to
+ * create a "keyless" experience for insert only workloads. The keys are guaranteed to be deterministic but not unique, so they can only be used for insert workflows with deduplication disabled.
+ * The class attempts to get sufficient uniqueness for keys to prevent frequent collisions by choosing the fields it uses in order of decreasing likelihood for uniqueness. The ordering is:
+ * <ul>
+ * <li>timestamp</li>
+ * <li>numeric values</li>
+ * <li>string, byte arrays, other types not mentioned</li>
+ * <li>date, lists, maps, booleans</li>
+ * </ul>
+ * The number of fields is capped to created predictable performance and the generator only uses non-null values to help increase uniqueness for sparse datasets.
+ */
+public class KeylessKeyGenerator extends CustomAvroKeyGenerator {
+ private static final String HOODIE_PREFIX = "_hoodie";
+ private static final String DOT = ".";
+ private final int maxFieldsToConsider;
+ private final int numFieldsForKey;
+ private final Set<String> partitionFieldNames;
+ private int[][] fieldOrdering;
+
+ public KeylessKeyGenerator(TypedProperties props) {
+ super(props);
+ this.numFieldsForKey = props.getInteger(KeyGeneratorOptions.NUM_FIELDS_IN_KEYLESS_GENERATOR.key(), KeyGeneratorOptions.NUM_FIELDS_IN_KEYLESS_GENERATOR.defaultValue());
+ // cap the number of fields to order in case of large schemas
+ this.maxFieldsToConsider = numFieldsForKey * 3;
+ this.partitionFieldNames = this.getPartitionPathFields().stream().map(field -> field.split(SPLIT_REGEX)[0]).collect(Collectors.toSet());
+ }
+
+ @Override
+ public String getRecordKey(GenericRecord record) {
+ return buildKey(getFieldOrdering(record), record);
+ }
+
+ int[][] getFieldOrdering(GenericRecord genericRecord) {
+ if (fieldOrdering == null) {
+ fieldOrdering = buildFieldOrdering(genericRecord.getSchema().getFields());
+ }
+ return fieldOrdering;
+ }
+
+ /**
+ * Deterministically builds a key for the input value based on the provided fieldOrdering. The first {@link #numFieldsForKey} non-null values will be used to generate a string that is passed to
+ * {@link UUID#nameUUIDFromBytes(byte[])}.
+ * @param fieldOrdering an array of integer arrays. The integer arrays represent paths to a single field within the input object.
+ * @param input the input object that needs a key
+ * @return a deterministically generated {@link UUID}
+ * @param <T> the input object type
+ */
+ private <T> String buildKey(int[][] fieldOrdering, GenericRecord input) {
+ StringBuilder key = new StringBuilder();
+ int nonNullFields = 0;
+ for (int[] index : fieldOrdering) {
+ Object value = getFieldForRecord(input, index);
+ if (value == null) {
+ continue;
+ }
+ nonNullFields++;
+ key.append(value.hashCode());
Review Comment:
Correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1382232963
hey @kazdy : we also jammed quite a bit before arriving at this solution. For eg, we did take a stab at generating unique Ids for every record [here](https://github.com/apache/hudi/pull/7622), but the problem as stated by Tim might not work for 7622. for eg, if we zoom into what happens for a commit in hudi is,
keyGen -> index look up -> upsert partitioner -> write files by executor (merge handle or create handle or append handle) -> may be write to metadata table -> complete commit.
Main crux here is that, in Upsert partitioner, we assign records to diff insert buckets based on record key hash. lets say upsert partitioned determined to add 3 new insert buckets and split 30k records among 3 insert bucket (file groups). This assignment is done using hashing of record key.
Given this, if due to failures, if keyGen stage was retriggered for a subset of spark partitions again, and when it reaches the upsert partitioner, it could get assigned to a diff insert bucket compared to its 1st attempt and so there are chances we will miss some records or add pack more records to one file group that what we intended.
Let me know if this makes sense. happy to jam to see if we can really pull this off by a row Id sort of generating rather than based on record payload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1380512238
CI failure due to unrelated flaky test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
kazdy commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1385496680
Thanks for the explanation, so it seems like key generator must be deterministic and there's no way around it.
What I do with hudi datasets where I need a surrogate key is that I just generate a column with UUID using [built-in spark uuid() function](https://spark.apache.org/docs/3.3.1/api/sql/#uuid). I think it's a valid way to do it :)
I guess using engine-specific uuid() function in keygen would not change anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7640: Add in support for a keyless workflow by building an ID based off of …
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1378142812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14232",
"triggerID" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea4b26959daa2f244c965b60101a1f0d09253394 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14232)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1408800988
> Let me know if this makes sense. happy to jam to see if we can really pull this off by a row Id sort of generating rather than based on record payload.
@nsivabalan I did some reading and found out that oracle and postgres both use pseudo/ system columns to imitate PK if not defined.
Do you think it would be possible to do something similar to oracle [ROWID](https://docs.oracle.com/database/121/SQLRF/sql_elements001.htm#SQLRF00213) pseudo column or postgres [ctid](https://www.postgresql.org/docs/current/ddl-system-columns.html) system column?
> Rowids contain the following information:
The data block of the data file containing the row. The length of this string depends on your operating system.
- The row in the data block.
- The database file containing the row. The first data file has the number 1. The length of this string depends on your operating system.
- The data object number, which is an identification number assigned to every database segment. You can retrieve the data object number from the data dictionary views USER_OBJECTS, DBA_OBJECTS, and ALL_OBJECTS. Objects that share the same segment (clustered tables in the same cluster, for example) have the same object number.
It seems like it would be doable with vectorized parquet reader rowId/ Column Vector etc. instead of "row in data block", the file name is known and saved in meta columns.
I don't know how would it handle the first write since there's no information about the column vector and hash can not be generated. So this can be impossible to use in an upsert partitioner and the whole idea does make any sense :) .
I see some restrictions:
- only support it in CoW (bc. parquet vectorized reader needs to be used?),
- only available with Virtual Keys,
- no incremental queries allowed (I think cdc from non-pk table is not supported in oracle rdbms),
- no support for datasource write with upsert when ROWID/ recordkey is not provided (should not be a problem with spark sql since it first queries Hudi and therefore it would be possible to get ROWID) (?)
But it would allow doing DS+SQL insert and SQL updates and SQL deletes without the need to define PK on the table.
Does it make any sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1379039866
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14232",
"triggerID" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"triggerType" : "PUSH"
}, {
"hash" : "70468d91500dff0bbc87701b3af5cfc3b8f6a6f8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70468d91500dff0bbc87701b3af5cfc3b8f6a6f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea4b26959daa2f244c965b60101a1f0d09253394 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14232)
* 70468d91500dff0bbc87701b3af5cfc3b8f6a6f8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
kazdy commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1380861070
Hi @the-other-tim-brown
I'm interested in this functionality and have some questions, if I understand correctly the UUID will be the same for the same set of values in columns that it's based on?
So this generator can't be used for generating a surrogate key (a standard practice in data warehousing) as key is derived from data? My understanding of keyless model is that record key is a surrogate key that's globally unique.
I'm wondering if there's something that does not allow to create globally unique ids via the key generator interface (maybe virtual keys support)?
At the same time in spite of this PR, what's the place of [UuidKeyGenerator](https://github.com/apache/hudi/blob/41a9986a7641f3232b1edd2a737fd4b7aa430dbf/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/UuidKeyGenerator.scala)? Could it be used to generate surrogate keys that are globally unique?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] the-other-tim-brown commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
the-other-tim-brown commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1382053922
> Hi @the-other-tim-brown I'm interested in this functionality and have some questions, if I understand correctly the UUID will be the same for the same set of values in columns that it's based on?
>
> So this generator can't be used for generating a surrogate key (a standard practice in data warehousing) as key is derived from data? My understanding of keyless model is that record key is a surrogate key that's globally unique.
>
> I'm wondering if there's something that does not allow to create globally unique ids via the key generator interface (maybe virtual keys support)? At the same time in context of this PR, what's the place of [UuidKeyGenerator](https://github.com/apache/hudi/blob/41a9986a7641f3232b1edd2a737fd4b7aa430dbf/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/UuidKeyGenerator.scala)? Could it be used to generate surrogate keys that are globally unique?
Yes it is correct that the keys are not guaranteed to be unique here. The issue with using a random UUID for us was that we were using deltastreamer and if the dag ever retriggered we were seeing data generated with new random UUIDs which could cause the records to be written to different filegroups causing an issue with duplicate/lost data due to some internals of how Hudi works. @nsivabalan had some similar thoughts around other approaches, can you chime in here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7640: Add in support for a keyless workflow by building an ID based off of …
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1378138384
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea4b26959daa2f244c965b60101a1f0d09253394 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1379667097
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14232",
"triggerID" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"triggerType" : "PUSH"
}, {
"hash" : "70468d91500dff0bbc87701b3af5cfc3b8f6a6f8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14253",
"triggerID" : "70468d91500dff0bbc87701b3af5cfc3b8f6a6f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 70468d91500dff0bbc87701b3af5cfc3b8f6a6f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14253)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1380517021
will go ahead and land this for now. will get to consensus async.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #7640:
URL: https://github.com/apache/hudi/pull/7640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7640: Add in support for a keyless workflow by building an ID based off of …
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1378339063
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14232",
"triggerID" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea4b26959daa2f244c965b60101a1f0d09253394 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14232)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7640: [HUDI-5514] Add in support for a keyless workflow
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7640:
URL: https://github.com/apache/hudi/pull/7640#issuecomment-1379059409
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14232",
"triggerID" : "ea4b26959daa2f244c965b60101a1f0d09253394",
"triggerType" : "PUSH"
}, {
"hash" : "70468d91500dff0bbc87701b3af5cfc3b8f6a6f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14253",
"triggerID" : "70468d91500dff0bbc87701b3af5cfc3b8f6a6f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea4b26959daa2f244c965b60101a1f0d09253394 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14232)
* 70468d91500dff0bbc87701b3af5cfc3b8f6a6f8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14253)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org