You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/09 03:24:31 UTC
[GitHub] [hudi] xiarixiaoyao opened a new pull request #3952: [HUDI-2102][WIP] support hilbert curve for hudi.
xiarixiaoyao opened a new pull request #3952:
URL: https://github.com/apache/hudi/pull/3952
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
support hilbert curve for hudi.
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r748838730
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
Review comment:
Thank you for your review。 may be i miss somethings， could you point out the problem more details thanks。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966124549
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a3305e70773578729177c6ee863d52ecf31ee39 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235)
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980466391
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d034fb15666134c98cfff743715ec0aeee29602 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980481068
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839)
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980502905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850)
* 18d2357845c09748e5cd7297b3d3f4569baad7d8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r756647750
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestTableLayoutOptimization.scala
##########
@@ -68,8 +66,8 @@ class TestTableLayoutOptimization extends HoodieClientTestBase {
}
@ParameterizedTest
- @ValueSource(strings = Array("COPY_ON_WRITE", "MERGE_ON_READ"))
- def testOptimizeWithClustering(tableType: String): Unit = {
+ @CsvSource(Array("COPY_ON_WRITE, hilbert", "COPY_ON_WRITE, z-order", "MERGE_ON_READ, hilbert", "MERGE_ON_READ, z-order"))
Review comment:
@alexeykudinkin could you pls help me， i donnot know how to use @MethodSource in scala。 thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-979037711
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4bd2f9aeab89bdd209db4d64b1f5a95696717c5d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744)
* d05366ef0fab06fab3a6ec283ab6b12069fbaba4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747)
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-979135993
@vinothchandar @alexeykudinkin @yihua addressed all comments. pls help me review those code again, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r754825800
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/ZOrderingUtil.java
##########
@@ -176,9 +176,17 @@ public static byte updatePos(byte a, int apos, byte b, int bpos) {
public static Long convertStringToLong(String a) {
Review comment:
sorry, let me add UT for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102][WIP] support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-963783951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a3305e70773578729177c6ee863d52ecf31ee39 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980442192
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749)
* cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980477105
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836)
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980473749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d034fb15666134c98cfff743715ec0aeee29602 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830)
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980473382
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d034fb15666134c98cfff743715ec0aeee29602 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830)
* 411029ac026d4a7fa528a486240837d775fe3a75 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980492697
@vinothchandar sorry， our company sealed the slack， now i cannot log in。
could you pls post the question directly here? Thank you very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980499225
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980475975
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836)
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua edited a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua edited a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980483457
> @yihua pls wait, after some fixed the benchmark cannot run let me check it
Got it. I'll let you check that. Let me know when it's good to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980507331
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852",
"triggerID" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850)
* 18d2357845c09748e5cd7297b3d3f4569baad7d8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980516914
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852",
"triggerID" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 18d2357845c09748e5cd7297b3d3f4569baad7d8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980475975
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836)
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r756627131
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java
##########
@@ -458,4 +458,27 @@ public static BuildCurveStrategyType fromValue(String value) {
}
}
}
+
+ /**
+ * strategy types for optimize layout for hudi data.
+ */
+ public enum BuildLayoutOptimizationStrategy {
+ ZORDER("z-order"),
+ HILBERT("hilbert");
+ private final String value;
+ BuildLayoutOptimizationStrategy(String value) {
+ this.value = value;
+ }
+
+ public static BuildLayoutOptimizationStrategy fromValue(String value) {
Review comment:
oh sorry,
i want use "z-order", "hilbert" as the parmeter not "ZORDER". "HILBERT"
and if we want to call this method in scala. valueOf will not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r757729509
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/spark/OrderingIndexHelper.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieColumnRangeMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.util.BaseFileUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.index.zorder.ZOrderingIndexHelper;
+import org.apache.hudi.optimize.HilbertCurveUtils;
+import org.apache.hudi.optimize.ZOrderingUtil;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.Row$;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.hudi.execution.RangeSampleSort$;
+import org.apache.spark.sql.hudi.execution.ZorderingBinarySort;
+import org.apache.spark.sql.types.BinaryType;
+import org.apache.spark.sql.types.BinaryType$;
+import org.apache.spark.sql.types.BooleanType;
+import org.apache.spark.sql.types.ByteType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DateType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.DoubleType;
+import org.apache.spark.sql.types.FloatType;
+import org.apache.spark.sql.types.IntegerType;
+import org.apache.spark.sql.types.LongType;
+import org.apache.spark.sql.types.LongType$;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.ShortType;
+import org.apache.spark.sql.types.StringType;
+import org.apache.spark.sql.types.StringType$;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType$;
+import org.apache.spark.sql.types.TimestampType;
+import org.apache.spark.util.SerializableConfiguration;
+import org.davidmoten.hilbert.HilbertCurve;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import scala.collection.JavaConversions;
+
+public class OrderingIndexHelper {
Review comment:
@xiarixiaoyao why is this not part of the `org.apache.hudi` package
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/spark/OrderingIndexHelper.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieColumnRangeMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.util.BaseFileUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.index.zorder.ZOrderingIndexHelper;
+import org.apache.hudi.optimize.HilbertCurveUtils;
+import org.apache.hudi.optimize.ZOrderingUtil;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.Row$;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.hudi.execution.RangeSampleSort$;
+import org.apache.spark.sql.hudi.execution.ZorderingBinarySort;
+import org.apache.spark.sql.types.BinaryType;
+import org.apache.spark.sql.types.BinaryType$;
+import org.apache.spark.sql.types.BooleanType;
+import org.apache.spark.sql.types.ByteType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DateType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.DoubleType;
+import org.apache.spark.sql.types.FloatType;
+import org.apache.spark.sql.types.IntegerType;
+import org.apache.spark.sql.types.LongType;
+import org.apache.spark.sql.types.LongType$;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.ShortType;
+import org.apache.spark.sql.types.StringType;
+import org.apache.spark.sql.types.StringType$;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType$;
+import org.apache.spark.sql.types.TimestampType;
+import org.apache.spark.util.SerializableConfiguration;
Review comment:
we have our own SerializableConfiguration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r755636337
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
+ X[2] = C F I L O axes |/
+ high low 0------> X[0]
+ * </pre>
+ *
+ * @param index
+ * index to be tranposed
+ * @return transposed index
+ */
+ long[] transpose(BigInteger index) {
+ long[] x = new long[dimensions];
+ transpose(index, x);
+ return x;
+ }
+
+ private void transpose(BigInteger index, long[] x) {
+ byte[] b = index.toByteArray();
+ for (int idx = 0; idx < 8 * b.length; idx++) {
+ if ((b[b.length - 1 - idx / 8] & (1L << (idx % 8))) != 0) {
+ int dim = (length - idx - 1) % dimensions;
+ int shift = (idx / dimensions) % bits;
+ x[dim] |= 1L << shift;
+ }
+ }
+ }
+
+ /**
+ * <p>
+ * Given the axes (coordinates) of a point in N-Dimensional space, find the
+ * distance to that point along the Hilbert curve. That distance will be
+ * transposed; broken into pieces and distributed into an array.
+ *
+ * <p>
+ * The number of dimensions is the length of the hilbertAxes array.
+ *
+ * <p>
+ * Note: In Skilling's paper, this function is called AxestoTranspose.
+ *
+ * @param bits
+ * @param point
+ * Point in N-space
Review comment:
Can you please take a look at taking care of java-doc formatting across the board?
My suggestions would be
- If there's no comment for an arg (for ex, in case it's trivial) -- let's skip the line with it altogether (much better than an empty doc)
- Keep the comment on the same line as parameter
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
+ X[2] = C F I L O axes |/
+ high low 0------> X[0]
+ * </pre>
+ *
+ * @param index
+ * index to be tranposed
+ * @return transposed index
+ */
+ long[] transpose(BigInteger index) {
+ long[] x = new long[dimensions];
+ transpose(index, x);
+ return x;
+ }
+
+ private void transpose(BigInteger index, long[] x) {
+ byte[] b = index.toByteArray();
+ for (int idx = 0; idx < 8 * b.length; idx++) {
+ if ((b[b.length - 1 - idx / 8] & (1L << (idx % 8))) != 0) {
+ int dim = (length - idx - 1) % dimensions;
+ int shift = (idx / dimensions) % bits;
+ x[dim] |= 1L << shift;
+ }
+ }
+ }
+
+ /**
+ * <p>
+ * Given the axes (coordinates) of a point in N-Dimensional space, find the
+ * distance to that point along the Hilbert curve. That distance will be
+ * transposed; broken into pieces and distributed into an array.
+ *
+ * <p>
+ * The number of dimensions is the length of the hilbertAxes array.
+ *
+ * <p>
+ * Note: In Skilling's paper, this function is called AxestoTranspose.
+ *
+ * @param bits
+ * @param point
+ * Point in N-space
+ * @return The Hilbert distance (or index) as a transposed Hilbert index
+ */
+ static long[] transposedIndex(int bits, long... point) {
Review comment:
Let's make sure we have tests for these utility methods
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
+ X[2] = C F I L O axes |/
+ high low 0------> X[0]
+ * </pre>
+ *
+ * @param index
+ * index to be tranposed
+ * @return transposed index
+ */
+ long[] transpose(BigInteger index) {
+ long[] x = new long[dimensions];
+ transpose(index, x);
+ return x;
+ }
+
+ private void transpose(BigInteger index, long[] x) {
+ byte[] b = index.toByteArray();
+ for (int idx = 0; idx < 8 * b.length; idx++) {
+ if ((b[b.length - 1 - idx / 8] & (1L << (idx % 8))) != 0) {
+ int dim = (length - idx - 1) % dimensions;
+ int shift = (idx / dimensions) % bits;
+ x[dim] |= 1L << shift;
+ }
+ }
+ }
+
+ /**
+ * <p>
+ * Given the axes (coordinates) of a point in N-Dimensional space, find the
+ * distance to that point along the Hilbert curve. That distance will be
+ * transposed; broken into pieces and distributed into an array.
+ *
+ * <p>
+ * The number of dimensions is the length of the hilbertAxes array.
+ *
+ * <p>
+ * Note: In Skilling's paper, this function is called AxestoTranspose.
+ *
+ * @param bits
+ * @param point
+ * Point in N-space
+ * @return The Hilbert distance (or index) as a transposed Hilbert index
+ */
+ static long[] transposedIndex(int bits, long... point) {
+ final long M = 1L << (bits - 1);
+ final int n = point.length; // n: Number of dimensions
+ final long[] x = Arrays.copyOf(point, n);
+ long p;
+ long q;
+ long t;
Review comment:
We can combine this 3 into one decl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966124549
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a3305e70773578729177c6ee863d52ecf31ee39 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235)
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r754786477
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/ZOrderingUtil.java
##########
@@ -176,9 +176,17 @@ public static byte updatePos(byte a, int apos, byte b, int bpos) {
public static Long convertStringToLong(String a) {
Review comment:
do we have UT for these?
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestOptimizeTable.scala
##########
@@ -64,8 +64,11 @@ class TestOptimizeTable extends HoodieClientTestBase {
}
@ParameterizedTest
- @ValueSource(strings = Array("COPY_ON_WRITE", "MERGE_ON_READ"))
- def testOptimizewithClustering(tableType: String): Unit = {
+ @ValueSource(strings = Array("COPY_ON_WRITE, hilbert", "COPY_ON_WRITE, z-order", "MERGE_ON_READ, hilbert", "MERGE_ON_READ, z-order"))
Review comment:
can we have two parameters? instead of the comma based splits.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
Review comment:
this probably needs to be added to the NOTICE? cc @leesf @yanghua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980491819
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 4d1777324ea0588cf68ee56c439610278e222192 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842)
* 503ebc7ff871791004511bae1bc79465e341062d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980493177
> @yihua i run benchmark， it's ok now. thanks very much
>
> for table table_z_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.785 for table table_z_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.76 for table table_hilbert_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.845 for table table_hilbert_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.83 for table table_z_sort_byMap_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.0 for table table_z_sort_bySample_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.78 for table table_hilbert_sort_byMap_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.07999999999999996 for table table_hilbert_sort_bySample_skew with query filter: c1_int = 5000 or c2_int = 500000 we can
achieve skipping percent 0.83
Great job @xiarixiaoyao on getting this done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r756672557
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/SpaceCurveOptimizeBenchMark.scala
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.SpaceCurveOptimizeHelper
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.hudi.TestHoodieSqlBase
+
+import scala.util.Random
+
+object SpaceCurveOptimizeBenchMark extends TestHoodieSqlBase {
+
+ def getSkippingPercent(tableName: String, co1: String, co2: String, value1: Int, value2: Int): Unit= {
+ val minMax = SpaceCurveOptimizeHelper
+ .getMinMaxValue(spark.sql(s"select * from ${tableName}"), s"${co1}, ${co2}")
+ .collect().map(f => (f.getInt(1), f.getInt(2), f.getInt(4), f.getInt(5)))
+ var c = 0
+ for (elem <- minMax) {
+ if ((elem._1 <= value1 && elem._2 >= value1) || (elem._3 <= value2 && elem._4 >= value2)) {
+ c = c + 1
+ }
+ }
+
+ val p = c / minMax.size.toDouble
+ println(s"for table ${tableName} with query filter: ${co1} = ${value1} or ${co2} = ${value2} we can achieve skipping percent ${1.0 - p}")
+ }
+
+ /*
+ for table table_z_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.8
+ for table table_z_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.77
+ for table table_hilbert_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.855
+ for table table_hilbert_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.83
+ */
+ def runNormalTableSkippingBenchMark(): Unit = {
+ withTempDir { f =>
+ withTempTable("table_z_sort_byMap", "table_z_sort_bySample", "table_hilbert_sort_byMap", "table_hilbert_sort_bySample") {
+ prepareNormalTable(new Path(f.getAbsolutePath), 1000000)
+ // choose median value as filter condition.
+ // the median value of c1_int is 500000
+ // the median value of c2_int is 500000
+ getSkippingPercent("table_z_sort_byMap", "c1_int", "c2_int", 500000, 500000)
+ getSkippingPercent("table_z_sort_bySample", "c1_int", "c2_int", 500000, 500000)
+ getSkippingPercent("table_hilbert_sort_byMap", "c1_int", "c2_int", 500000, 500000)
+ getSkippingPercent("table_hilbert_sort_bySample", "c1_int", "c2_int", 500000, 500000)
+ }
+ }
+ }
+
+ /*
+ for table table_z_sort_byMap_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.0
+ for table table_z_sort_bySample_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.78
+ for table table_hilbert_sort_byMap_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.05500000000000005
+ for table table_hilbert_sort_bySample_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.84
+ */
+ def runSkewTableSkippingBenchMark(): Unit = {
+ withTempDir { f =>
+ withTempTable("table_z_sort_byMap_skew", "table_z_sort_bySample_skew", "table_hilbert_sort_byMap_skew", "table_hilbert_sort_bySample_skew") {
+ prepareSkewTable(new Path(f.getAbsolutePath), 1000000)
+ // choose median value as filter condition.
+ // the median value of c1_int is 5000
+ // the median value of c2_int is 500000
+ getSkippingPercent("table_z_sort_byMap_skew", "c1_int", "c2_int", 5000, 500000)
+ getSkippingPercent("table_z_sort_bySample_skew", "c1_int", "c2_int", 5000, 500000)
+ getSkippingPercent("table_hilbert_sort_byMap_skew", "c1_int", "c2_int", 5000, 500000)
+ getSkippingPercent("table_hilbert_sort_bySample_skew", "c1_int", "c2_int", 5000, 500000)
+ }
+ }
+ }
+
+ def main(args: Array[String]): Unit = {
+ runNormalTableSkippingBenchMark()
+ runSkewTableSkippingBenchMark()
+ }
+
+ def withTempTable(tableNames: String*)(f: => Unit): Unit = {
+ try f finally tableNames.foreach(spark.catalog.dropTempView)
+ }
+
+ def prepareNormalTable(tablePath: Path, numRows: Int): Unit = {
+ import spark.implicits._
+ val df = spark.range(numRows).map(_ => (Random.nextInt(1000000), Random.nextInt(1000000))).toDF("c1_int", "c2_int")
+ val dfOptimizeByMap = SpaceCurveOptimizeHelper.createOptimizedDataFrameByMapValue(df, "c1_int, c2_int", 200, "z-order")
+ val dfOptimizeBySample = SpaceCurveOptimizeHelper.createOptimizeDataFrameBySample(df, "c1_int, c2_int", 200, "z-order")
+
+ val dfHilbertOptimizeByMap = SpaceCurveOptimizeHelper.createOptimizedDataFrameByMapValue(df, "c1_int, c2_int", 200, "hilbert")
+ val dfHilbertOptimizeBySample = SpaceCurveOptimizeHelper.createOptimizeDataFrameBySample(df, "c1_int, c2_int", 200, "hilbert")
+
+ saveAsTable(dfOptimizeByMap, tablePath, "z_sort_byMap")
+ saveAsTable(dfOptimizeBySample, tablePath, "z_sort_bySample")
+ saveAsTable(dfHilbertOptimizeByMap, tablePath, "hilbert_sort_byMap")
+ saveAsTable(dfHilbertOptimizeBySample, tablePath, "hilbert_sort_bySample")
+ }
+
+ def prepareSkewTable(tablePath: Path, numRows: Int): Unit = {
Review comment:
fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-979037711
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4bd2f9aeab89bdd209db4d64b1f5a95696717c5d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744)
* d05366ef0fab06fab3a6ec283ab6b12069fbaba4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747)
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978981419
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4bd2f9aeab89bdd209db4d64b1f5a95696717c5d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744)
* d05366ef0fab06fab3a6ec283ab6b12069fbaba4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-979131121
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r757696330
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/spark/SpaceCurveOptimizeHelper.java
##########
@@ -67,40 +69,62 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
-public class ZCurveOptimizeHelper {
+public class SpaceCurveOptimizeHelper {
private static final String SPARK_JOB_DESCRIPTION = "spark.job.description";
/**
- * Create z-order DataFrame directly
- * first, map all base type data to byte[8], then create z-order DataFrame
+ * Create optimized DataFrame directly
* only support base type data. long,int,short,double,float,string,timestamp,decimal,date,byte
- * this method is more effective than createZIndexDataFrameBySample
+ * this method is more effective than createOptimizeDataFrameBySample
*
* @param df a spark DataFrame holds parquet files to be read.
- * @param zCols z-sort cols
+ * @param sortCols z-sort/hilbert-sort cols
* @param fileNum spark partition num
- * @return a dataFrame sorted by z-order.
+ * @param sortMode layout optimization strategy
+ * @return a dataFrame sorted by z-order/hilbert.
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r757696289
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/spark/SpaceCurveOptimizeHelper.java
##########
@@ -67,40 +69,62 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
-public class ZCurveOptimizeHelper {
+public class SpaceCurveOptimizeHelper {
private static final String SPARK_JOB_DESCRIPTION = "spark.job.description";
/**
- * Create z-order DataFrame directly
- * first, map all base type data to byte[8], then create z-order DataFrame
+ * Create optimized DataFrame directly
* only support base type data. long,int,short,double,float,string,timestamp,decimal,date,byte
- * this method is more effective than createZIndexDataFrameBySample
+ * this method is more effective than createOptimizeDataFrameBySample
*
* @param df a spark DataFrame holds parquet files to be read.
- * @param zCols z-sort cols
+ * @param sortCols z-sort/hilbert-sort cols
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980501699
@yihua fixed the issue。 waiting for UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980466391
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d034fb15666134c98cfff743715ec0aeee29602 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980487292
@vinothchandar @yihua . can we still use the orginal code instead of use com.github.davidmoten: the Hilbert curve package directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980505584
@yihua
This is about the logic of saving statistics, which is normal. hilbert and zorder will use the same logical to save statistic info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-976192499
@vinothchandar @alexeykudinkin @yanghua thanks for your review. i will try to adressed all comments
_however As more and more columns participate in sorting, Hilbert will be slower and slower。
you mean to build or the skipping efficiency?_
answer: build will be slower, not skipping efficiency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980487801
@xiarixiaoyao We already updated NOTICE. So we can revert back to that.
I am debugging some issue with z-ordering on a larger dataset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980451466
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824)
* 0d034fb15666134c98cfff743715ec0aeee29602 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980448546
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980487983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840)
* 4d1777324ea0588cf68ee56c439610278e222192 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980490238
> @yihua could you pls help me，now i cannot pull/push any code in github。 those are my modify： patch1： modify HilbertCurveUitls public class HilbertCurveUtils { public static byte[] indexBytes(HilbertCurve hilbertCurve, long[] points, int paddingNum) { BigInteger index = hilbertCurve.index(points); return paddingToNByte(index.toByteArray(), paddingNum); }
>
> public static byte[] paddingToNByte(byte[] a, int paddingNum) { if (a.length == paddingNum) { return a; } if (a.length > paddingNum) { byte[] result = new byte[paddingNum]; System.arraycopy(a, 0, result, 0, paddingNum); return result; } int paddingSize = 8 - a.length; byte[] result = new byte[paddingNum]; for (int i = 0; i < paddingSize; i++) { result[i] = 0; } System.arraycopy(a, 0, result, paddingSize, a.length); return result; } }
>
> patch2： line220 in OrderingIndexHelper: hilbertCurve, longList.stream().mapToLong(l -> l).toArray(), 63);
>
> patch3: line520 in RangeSample: HilbertCurveUtils.indexBytes(hilbertCurve.get, values.map(_.toLong).toArray, 32)
>
> patch4: line 99 in SpaceCurveOptimizeBenchMark: val df = spark.range(numRows).map(_ => (Random.nextInt(col1Range), Random.nextInt(col2Range))).toDF("c1_int", "c2_int")
Let me push a commit for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980507331
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852",
"triggerID" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850)
* 18d2357845c09748e5cd7297b3d3f4569baad7d8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980493766
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 4d1777324ea0588cf68ee56c439610278e222192 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842)
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980505736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852",
"triggerID" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850)
* 18d2357845c09748e5cd7297b3d3f4569baad7d8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980474439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102][WIP] support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-963784924
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a3305e70773578729177c6ee863d52ecf31ee39 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966119536
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a3305e70773578729177c6ee863d52ecf31ee39 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235)
* 60c291565f976aa39d5e94591ab58079230c2358 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966175269
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-963806283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a3305e70773578729177c6ee863d52ecf31ee39 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978835526
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307)
* 8a2c78442579ffd076f752f4b65d41778e482549 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r756615237
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java
##########
@@ -458,4 +458,27 @@ public static BuildCurveStrategyType fromValue(String value) {
}
}
}
+
+ /**
+ * strategy types for optimize layout for hudi data.
+ */
+ public enum BuildLayoutOptimizationStrategy {
+ ZORDER("z-order"),
+ HILBERT("hilbert");
+ private final String value;
+ BuildLayoutOptimizationStrategy(String value) {
+ this.value = value;
+ }
+
+ public static BuildLayoutOptimizationStrategy fromValue(String value) {
Review comment:
SpaceCurveOptimizeHelper.createOptimizedDataFrameByMapValue will use this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978955941
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45e4d2175d249bc68889b807a1833ac6fac434c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739)
* 4bd2f9aeab89bdd209db4d64b1f5a95696717c5d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978958533
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45e4d2175d249bc68889b807a1833ac6fac434c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739)
* 4bd2f9aeab89bdd209db4d64b1f5a95696717c5d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978836367
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307)
* 8a2c78442579ffd076f752f4b65d41778e482549 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-979084670
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d05366ef0fab06fab3a6ec283ab6b12069fbaba4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747)
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-979039987
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d05366ef0fab06fab3a6ec283ab6b12069fbaba4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747)
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980477105
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836)
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980481836
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840)
* 4d1777324ea0588cf68ee56c439610278e222192 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980481373
> @yihua Sorry, due to the time difference, I only see these comments now. Thank you very much for your help
No worries. If you could help take another pass of the code change to see if I miss anything, that'd be great. There're changes from master around Z-ordering and I resolved the rebasing conflicts. I'm addressing dependency issues in the bundles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980486706
for table table_z_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.795
for table table_z_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.76
for table table_hilbert_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.18999999999999995
for table table_hilbert_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.21999999999999997
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r757629889
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
Review comment:
Is this class copied from https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java? Could we just add that library as a dependency and have a wrapper class around it if needed?
```
<dependency>
<groupId>com.github.davidmoten</groupId>
<artifactId>hilbert-curve</artifactId>
<version>VERSION_HERE</version>
</dependency>
```
##########
File path: hudi-client/hudi-client-common/src/test/java/org/apache/hudi/optimize/TestZOrderingUtil.java
##########
@@ -126,4 +126,21 @@ public OrginValueWrapper(T index, T originValue) {
this.originValue = originValue;
}
}
+
+ @Test
+ public void testConvertBytesToLong() {
Review comment:
Could you add another test for the cases when the length of the byte array passed to `convertLongToBytes()` is not 8, where padding logic is incurred?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/ZOrderingUtil.java
##########
@@ -176,9 +176,17 @@ public static byte updatePos(byte a, int apos, byte b, int bpos) {
public static Long convertStringToLong(String a) {
byte[] bytes = utf8To8Byte(a);
+ return convertBytesToLong(bytes);
+ }
+
+ public static long convertBytesToLong(byte[] bytes) {
+ byte[] padBytes = bytes;
Review comment:
nit: can be named as `paddedBytes`
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/spark/SpaceCurveOptimizeHelper.java
##########
@@ -67,40 +69,62 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
-public class ZCurveOptimizeHelper {
+public class SpaceCurveOptimizeHelper {
private static final String SPARK_JOB_DESCRIPTION = "spark.job.description";
/**
- * Create z-order DataFrame directly
- * first, map all base type data to byte[8], then create z-order DataFrame
+ * Create optimized DataFrame directly
* only support base type data. long,int,short,double,float,string,timestamp,decimal,date,byte
- * this method is more effective than createZIndexDataFrameBySample
+ * this method is more effective than createOptimizeDataFrameBySample
*
* @param df a spark DataFrame holds parquet files to be read.
- * @param zCols z-sort cols
+ * @param sortCols z-sort/hilbert-sort cols
* @param fileNum spark partition num
- * @return a dataFrame sorted by z-order.
+ * @param sortMode layout optimization strategy
+ * @return a dataFrame sorted by z-order/hilbert.
Review comment:
similar here.
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/spark/SpaceCurveOptimizeHelper.java
##########
@@ -67,40 +69,62 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
-public class ZCurveOptimizeHelper {
+public class SpaceCurveOptimizeHelper {
private static final String SPARK_JOB_DESCRIPTION = "spark.job.description";
/**
- * Create z-order DataFrame directly
- * first, map all base type data to byte[8], then create z-order DataFrame
+ * Create optimized DataFrame directly
* only support base type data. long,int,short,double,float,string,timestamp,decimal,date,byte
- * this method is more effective than createZIndexDataFrameBySample
+ * this method is more effective than createOptimizeDataFrameBySample
*
* @param df a spark DataFrame holds parquet files to be read.
- * @param zCols z-sort cols
+ * @param sortCols z-sort/hilbert-sort cols
Review comment:
nit: `z-sort/hilbert-sort cols` -> `sorting columns`? (no need to mention sorting mechanism here, to be general)
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/ZOrderingUtil.java
##########
@@ -176,9 +176,17 @@ public static byte updatePos(byte a, int apos, byte b, int bpos) {
public static Long convertStringToLong(String a) {
byte[] bytes = utf8To8Byte(a);
+ return convertBytesToLong(bytes);
+ }
+
+ public static long convertBytesToLong(byte[] bytes) {
+ byte[] padBytes = bytes;
+ if (bytes.length != 8) {
+ padBytes = paddingTo8Byte(bytes);
+ }
Review comment:
you can simply have `byte[] paddedBytes = paddingTo8Byte(bytes);` since inside `paddingTo8Byte()` there is already check for the length.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980473382
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d034fb15666134c98cfff743715ec0aeee29602 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830)
* 411029ac026d4a7fa528a486240837d775fe3a75 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r757695776
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
Review comment:
I added the maven dependency of hilbert-curve and reused the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r757695972
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/ZOrderingUtil.java
##########
@@ -176,9 +176,17 @@ public static byte updatePos(byte a, int apos, byte b, int bpos) {
public static Long convertStringToLong(String a) {
byte[] bytes = utf8To8Byte(a);
+ return convertBytesToLong(bytes);
+ }
+
+ public static long convertBytesToLong(byte[] bytes) {
+ byte[] padBytes = bytes;
+ if (bytes.length != 8) {
+ padBytes = paddingTo8Byte(bytes);
+ }
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980500631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844)
* 18d2357845c09748e5cd7297b3d3f4569baad7d8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #3952: [HUDI-2102] Support hilbert curve for hudi
Posted by GitBox <gi...@apache.org>.
yihua merged pull request #3952:
URL: https://github.com/apache/hudi/pull/3952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102][WIP] support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-963784924
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a3305e70773578729177c6ee863d52ecf31ee39 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966286029
@vinothchandar could you help me review this pr, thanks.
test:
step1: prepare a parquet table wich has 136666612 rows, and 7 columns.
add col1 and col2 to this parquet table.
_**_val xs = df.schema.add("id1", IntegerType, true).add("id2", IntegerType)
val rdd = df.rdd.map { x =>
Row.fromSeq(x.toSeq ++ Seq(scala.util.Random.nextInt(1000000), scala.util.Random.nextInt(1000000)))
}
spark.createDataFrame(rdd, xs)_**_
col1, col2 are produced by scala.util.Random.nextInt(1000000)
step2: bulk insert a new hudi table
step3: do optimize the hudi table by sort col1,col2 with z-order/hilbert
step4: select count(*) from htest where id1 > 500000 and id2 > 500000;
--executor-cores 3 --num-executors 2 --executor-memory 8g
<html xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:w="urn:schemas-microsoft-com:office:word"
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml"
xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta name=ProgId content=Word.Document>
<meta name=Generator content="Microsoft Word 15">
<meta name=Originator content="Microsoft Word 15">
<link rel=File-List
href="file:///C:\Users\m00443775\AppData\Local\Temp\msohtmlclip1\01\clip_filelist.xml">
<!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:RelyOnVML/>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<link rel=themeData
href="file:///C:\Users\m00443775\AppData\Local\Temp\msohtmlclip1\01\clip_themedata.thmx">
<link rel=colorSchemeMapping
href="file:///C:\Users\m00443775\AppData\Local\Temp\msohtmlclip1\01\clip_colorschememapping.xml">
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:DrawingGridVerticalSpacing>7.8 磅</w:DrawingGridVerticalSpacing>
<w:DisplayHorizontalDrawingGridEvery>0</w:DisplayHorizontalDrawingGridEvery>
<w:DisplayVerticalDrawingGridEvery>2</w:DisplayVerticalDrawingGridEvery>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>ZH-CN</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:SpaceForUL/>
<w:BalanceSingleByteDoubleByteWidth/>
<w:DoNotLeaveBackslashAlone/>
<w:ULTrailSpace/>
<w:DoNotExpandShiftReturn/>
<w:AdjustLineHeightInTable/>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
<w:UseFELayout/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false"
DefSemiHidden="false" DefQFormat="false" DefPriority="99"
LatentStyleCount="371">
<w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 9"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="header"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footer"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index heading"/>
<w:LsdException Locked="false" Priority="35" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of figures"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope return"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="line number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="page number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of authorities"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="macro"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="toa heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 5"/>
<w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Closing"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Signature"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="true"
UnhideWhenUsed="true" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Message Header"/>
<w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Salutation"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Date"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Block Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="FollowedHyperlink"/>
<w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Document Map"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Plain Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="E-mail Signature"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Top of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Bottom of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal (Web)"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Acronym"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Cite"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Code"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Definition"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Keyboard"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Preformatted"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Sample"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Typewriter"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Variable"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Table"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation subject"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="No List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Contemporary"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Elegant"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Professional"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Balloon Text"/>
<w:LsdException Locked="false" Priority="0" Name="Table Grid"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Theme"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" QFormat="true"
Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" QFormat="true"
Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" QFormat="true"
Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" QFormat="true"
Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" QFormat="true"
Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" QFormat="true"
Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" SemiHidden="true"
UnhideWhenUsed="true" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/>
<w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/>
<w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/>
<w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/>
<w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/>
<w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/>
<w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/>
<w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 6"/>
</w:LatentStyles>
</xml><![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;
mso-font-alt:SimSun;
mso-font-charset:134;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:3 680460288 22 0 262145 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:variable;
mso-font-signature:-536869121 1107305727 33554432 0 415 0;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;
mso-font-charset:134;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:3 680460288 22 0 262145 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0cm;
margin-bottom:.0001pt;
line-height:150%;
mso-pagination:none;
mso-layout-grid-align:none;
text-autospace:none;
font-size:10.5pt;
font-family:"Times New Roman",serif;
mso-fareast-font-family:宋体;
layout-grid-mode:line;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-size:10.0pt;
mso-ansi-font-size:10.0pt;
mso-bidi-font-size:10.0pt;
mso-ascii-font-family:"Times New Roman";
mso-fareast-font-family:宋体;
mso-hansi-font-family:"Times New Roman";
mso-font-kerning:0pt;}
/* Page Definitions */
@page
{mso-page-border-surround-header:no;
mso-page-border-surround-footer:no;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;
mso-header-margin:36.0pt;
mso-footer-margin:36.0pt;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:普通表格;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman",serif;}
table.MsoTableGridLight
{mso-style-name:网格型浅色;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-priority:40;
mso-style-unhide:no;
border:solid #BFBFBF 1.0pt;
mso-border-themecolor:background1;
mso-border-themeshade:191;
mso-border-alt:solid #BFBFBF .5pt;
mso-border-themecolor:background1;
mso-border-themeshade:191;
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-border-insideh:.5pt solid #BFBFBF;
mso-border-insideh-themecolor:background1;
mso-border-insideh-themeshade:191;
mso-border-insidev:.5pt solid #BFBFBF;
mso-border-insidev-themecolor:background1;
mso-border-insidev-themeshade:191;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman",serif;}
</style>
<![endif]-->
</head>
<body lang=ZH-CN style='tab-interval:21.0pt;text-justify-trim:punctuation'>
<!--StartFragment-->
Table | File nums | Data size | query | DataSkipping
-- | -- | -- | -- | --
hilbert | 200 | 136666612 | 3686 ms | Total file size is: 200, after file skip size is: 57 skipping percent 0.715
z-order | 200 | 136666612 | 4573 ms | Total file size is: 200, after file skip size is: 67 skipping percent 0.665
z-order (use sample strategy ) | 200 | 136666612 | 3520 ms | Total file size is: 200, after file skip size is: 58 skipping percent 0.71
<!--EndFragment-->
</body>
</html>
hilbert has better optimization effect。
and use sample-optimize-strategy has a better optimization effect than use default optimize strategy.
however As more and more columns participate in sorting, Hilbert will be slower and slower。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980441485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749)
* cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980487983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840)
* 4d1777324ea0588cf68ee56c439610278e222192 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980488047
> @vinothchandar @yihua . can we still use the orginal code instead of use com.github.davidmoten: the Hilbert curve package directly
I think if that library has some performance issue we have to use your original code. But I might know the difference between your code vs the `com.github.davidmoten:hilbert-curve` lib. So in the current PR, I have the following conversion, which generates the byte array to put into the range:
```
public static byte[] indexBytes(HilbertCurve hilbertCurve, long[] points) {
BigInteger index = hilbertCurve.index(points);
return index.toByteArray();
}
```
instead of directly returning the internal byte array (currently it's internal byte array -> BigInteger -> BigInteger.toByteArray()). And I guess maybe BigInteger.toByteArray() messes things up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao edited a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980486706
for table table_z_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent **0.795**
for table table_z_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent **0.76**
for table table_hilbert_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent **0.18999999999999995**
for table table_hilbert_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent **0.21999999999999997**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978954164
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45e4d2175d249bc68889b807a1833ac6fac434c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-979039987
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d05366ef0fab06fab3a6ec283ab6b12069fbaba4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747)
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980480518
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r756631619
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/SpaceCurveOptimizeBenchMark.scala
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.SpaceCurveOptimizeHelper
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.hudi.TestHoodieSqlBase
+
+import scala.util.Random
+
+object SpaceCurveOptimizeBenchMark extends TestHoodieSqlBase {
+
+ def getSkippingPercent(tableName: String, co1: String, co2: String, value1: Int, value2: Int): Unit= {
+ val minMax = SpaceCurveOptimizeHelper
+ .getMinMaxValue(spark.sql(s"select * from ${tableName}"), s"${co1}, ${co2}")
+ .collect().map(f => (f.getInt(1), f.getInt(2), f.getInt(4), f.getInt(5)))
+ var c = 0
+ for (elem <- minMax) {
+ if ((elem._1 <= value1 && elem._2 >= value1) || (elem._3 <= value2 && elem._4 >= value2)) {
+ c = c + 1
+ }
+ }
+
+ val p = c / minMax.size.toDouble
+ println(s"for table ${tableName} with query filter: ${co1} = ${value1} or ${co2} = ${value2} we can achieve skipping percent ${1.0 - p}")
+ }
+
+ /*
+ for table table_z_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.8
+ for table table_z_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.77
+ for table table_hilbert_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.855
+ for table table_hilbert_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.83
+ */
+ def runNormalTableSkippingBenchMark(): Unit = {
+ withTempDir { f =>
+ withTempTable("table_z_sort_byMap", "table_z_sort_bySample", "table_hilbert_sort_byMap", "table_hilbert_sort_bySample") {
+ prepareNormalTable(new Path(f.getAbsolutePath), 1000000)
+ // choose median value as filter condition.
+ // the median value of c1_int is 500000
+ // the median value of c2_int is 500000
+ getSkippingPercent("table_z_sort_byMap", "c1_int", "c2_int", 500000, 500000)
+ getSkippingPercent("table_z_sort_bySample", "c1_int", "c2_int", 500000, 500000)
+ getSkippingPercent("table_hilbert_sort_byMap", "c1_int", "c2_int", 500000, 500000)
+ getSkippingPercent("table_hilbert_sort_bySample", "c1_int", "c2_int", 500000, 500000)
+ }
+ }
+ }
+
+ /*
+ for table table_z_sort_byMap_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.0
+ for table table_z_sort_bySample_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.78
+ for table table_hilbert_sort_byMap_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.05500000000000005
+ for table table_hilbert_sort_bySample_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.84
+ */
+ def runSkewTableSkippingBenchMark(): Unit = {
+ withTempDir { f =>
+ withTempTable("table_z_sort_byMap_skew", "table_z_sort_bySample_skew", "table_hilbert_sort_byMap_skew", "table_hilbert_sort_bySample_skew") {
+ prepareSkewTable(new Path(f.getAbsolutePath), 1000000)
+ // choose median value as filter condition.
+ // the median value of c1_int is 5000
+ // the median value of c2_int is 500000
+ getSkippingPercent("table_z_sort_byMap_skew", "c1_int", "c2_int", 5000, 500000)
+ getSkippingPercent("table_z_sort_bySample_skew", "c1_int", "c2_int", 5000, 500000)
+ getSkippingPercent("table_hilbert_sort_byMap_skew", "c1_int", "c2_int", 5000, 500000)
+ getSkippingPercent("table_hilbert_sort_bySample_skew", "c1_int", "c2_int", 5000, 500000)
+ }
+ }
+ }
+
+ def main(args: Array[String]): Unit = {
+ runNormalTableSkippingBenchMark()
+ runSkewTableSkippingBenchMark()
+ }
+
+ def withTempTable(tableNames: String*)(f: => Unit): Unit = {
+ try f finally tableNames.foreach(spark.catalog.dropTempView)
+ }
+
+ def prepareNormalTable(tablePath: Path, numRows: Int): Unit = {
+ import spark.implicits._
+ val df = spark.range(numRows).map(_ => (Random.nextInt(1000000), Random.nextInt(1000000))).toDF("c1_int", "c2_int")
+ val dfOptimizeByMap = SpaceCurveOptimizeHelper.createOptimizedDataFrameByMapValue(df, "c1_int, c2_int", 200, "z-order")
+ val dfOptimizeBySample = SpaceCurveOptimizeHelper.createOptimizeDataFrameBySample(df, "c1_int, c2_int", 200, "z-order")
+
+ val dfHilbertOptimizeByMap = SpaceCurveOptimizeHelper.createOptimizedDataFrameByMapValue(df, "c1_int, c2_int", 200, "hilbert")
+ val dfHilbertOptimizeBySample = SpaceCurveOptimizeHelper.createOptimizeDataFrameBySample(df, "c1_int, c2_int", 200, "hilbert")
+
+ saveAsTable(dfOptimizeByMap, tablePath, "z_sort_byMap")
+ saveAsTable(dfOptimizeBySample, tablePath, "z_sort_bySample")
+ saveAsTable(dfHilbertOptimizeByMap, tablePath, "hilbert_sort_byMap")
+ saveAsTable(dfHilbertOptimizeBySample, tablePath, "hilbert_sort_bySample")
+ }
+
+ def prepareSkewTable(tablePath: Path, numRows: Int): Unit = {
Review comment:
ok， will do it。
but
this class is a benchmark class， not a UT test。 This is just a sample class showing performance improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r756632291
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestTableLayoutOptimization.scala
##########
@@ -68,8 +66,8 @@ class TestTableLayoutOptimization extends HoodieClientTestBase {
}
@ParameterizedTest
- @ValueSource(strings = Array("COPY_ON_WRITE", "MERGE_ON_READ"))
- def testOptimizeWithClustering(tableType: String): Unit = {
+ @CsvSource(Array("COPY_ON_WRITE, hilbert", "COPY_ON_WRITE, z-order", "MERGE_ON_READ, hilbert", "MERGE_ON_READ, z-order"))
Review comment:
ok， will do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980491819
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 4d1777324ea0588cf68ee56c439610278e222192 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842)
* 503ebc7ff871791004511bae1bc79465e341062d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980492886
@yihua i run benchmark， it's ok now. thanks very much
for table table_z_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.785
for table table_z_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.76
for table table_hilbert_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.845
for table table_hilbert_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.83
for table table_z_sort_byMap_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.0
for table table_z_sort_bySample_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.78
for table table_hilbert_sort_byMap_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.07999999999999996
for table table_hilbert_sort_bySample_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.83
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980492296
@yanghua pls wait, i am run benchmark. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978835526
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307)
* 8a2c78442579ffd076f752f4b65d41778e482549 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978867417
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a2c78442579ffd076f752f4b65d41778e482549 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-979131121
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980441485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749)
* cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102][WIP] support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-963783951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a3305e70773578729177c6ee863d52ecf31ee39 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-976141742
>however As more and more columns participate in sorting, Hilbert will be slower and slower。
you mean to build or the skipping efficiency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966212378
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r754825653
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
Review comment:
oh, sorry let me fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r756628229
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestTableLayoutOptimization.scala
##########
@@ -68,8 +66,8 @@ class TestTableLayoutOptimization extends HoodieClientTestBase {
}
@ParameterizedTest
- @ValueSource(strings = Array("COPY_ON_WRITE", "MERGE_ON_READ"))
- def testOptimizeWithClustering(tableType: String): Unit = {
+ @CsvSource(Array("COPY_ON_WRITE, hilbert", "COPY_ON_WRITE, z-order", "MERGE_ON_READ, hilbert", "MERGE_ON_READ, z-order"))
Review comment:
Instead of composing this product manually, please use `@MethodSource` and compose it through 2 parameters
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/execution/benchmark/SpaceCurveOptimizeBenchMark.scala
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.hadoop.fs.Path
+import org.apache.spark.SpaceCurveOptimizeHelper
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.hudi.TestHoodieSqlBase
+
+import scala.util.Random
+
+object SpaceCurveOptimizeBenchMark extends TestHoodieSqlBase {
+
+ def getSkippingPercent(tableName: String, co1: String, co2: String, value1: Int, value2: Int): Unit= {
+ val minMax = SpaceCurveOptimizeHelper
+ .getMinMaxValue(spark.sql(s"select * from ${tableName}"), s"${co1}, ${co2}")
+ .collect().map(f => (f.getInt(1), f.getInt(2), f.getInt(4), f.getInt(5)))
+ var c = 0
+ for (elem <- minMax) {
+ if ((elem._1 <= value1 && elem._2 >= value1) || (elem._3 <= value2 && elem._4 >= value2)) {
+ c = c + 1
+ }
+ }
+
+ val p = c / minMax.size.toDouble
+ println(s"for table ${tableName} with query filter: ${co1} = ${value1} or ${co2} = ${value2} we can achieve skipping percent ${1.0 - p}")
+ }
+
+ /*
+ for table table_z_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.8
+ for table table_z_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.77
+ for table table_hilbert_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.855
+ for table table_hilbert_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.83
+ */
+ def runNormalTableSkippingBenchMark(): Unit = {
+ withTempDir { f =>
+ withTempTable("table_z_sort_byMap", "table_z_sort_bySample", "table_hilbert_sort_byMap", "table_hilbert_sort_bySample") {
+ prepareNormalTable(new Path(f.getAbsolutePath), 1000000)
+ // choose median value as filter condition.
+ // the median value of c1_int is 500000
+ // the median value of c2_int is 500000
+ getSkippingPercent("table_z_sort_byMap", "c1_int", "c2_int", 500000, 500000)
+ getSkippingPercent("table_z_sort_bySample", "c1_int", "c2_int", 500000, 500000)
+ getSkippingPercent("table_hilbert_sort_byMap", "c1_int", "c2_int", 500000, 500000)
+ getSkippingPercent("table_hilbert_sort_bySample", "c1_int", "c2_int", 500000, 500000)
+ }
+ }
+ }
+
+ /*
+ for table table_z_sort_byMap_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.0
+ for table table_z_sort_bySample_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.78
+ for table table_hilbert_sort_byMap_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.05500000000000005
+ for table table_hilbert_sort_bySample_skew with query filter: c1_int = 5000 or c2_int = 500000 we can achieve skipping percent 0.84
+ */
+ def runSkewTableSkippingBenchMark(): Unit = {
+ withTempDir { f =>
+ withTempTable("table_z_sort_byMap_skew", "table_z_sort_bySample_skew", "table_hilbert_sort_byMap_skew", "table_hilbert_sort_bySample_skew") {
+ prepareSkewTable(new Path(f.getAbsolutePath), 1000000)
+ // choose median value as filter condition.
+ // the median value of c1_int is 5000
+ // the median value of c2_int is 500000
+ getSkippingPercent("table_z_sort_byMap_skew", "c1_int", "c2_int", 5000, 500000)
+ getSkippingPercent("table_z_sort_bySample_skew", "c1_int", "c2_int", 5000, 500000)
+ getSkippingPercent("table_hilbert_sort_byMap_skew", "c1_int", "c2_int", 5000, 500000)
+ getSkippingPercent("table_hilbert_sort_bySample_skew", "c1_int", "c2_int", 5000, 500000)
+ }
+ }
+ }
+
+ def main(args: Array[String]): Unit = {
+ runNormalTableSkippingBenchMark()
+ runSkewTableSkippingBenchMark()
+ }
+
+ def withTempTable(tableNames: String*)(f: => Unit): Unit = {
+ try f finally tableNames.foreach(spark.catalog.dropTempView)
+ }
+
+ def prepareNormalTable(tablePath: Path, numRows: Int): Unit = {
+ import spark.implicits._
+ val df = spark.range(numRows).map(_ => (Random.nextInt(1000000), Random.nextInt(1000000))).toDF("c1_int", "c2_int")
+ val dfOptimizeByMap = SpaceCurveOptimizeHelper.createOptimizedDataFrameByMapValue(df, "c1_int, c2_int", 200, "z-order")
+ val dfOptimizeBySample = SpaceCurveOptimizeHelper.createOptimizeDataFrameBySample(df, "c1_int, c2_int", 200, "z-order")
+
+ val dfHilbertOptimizeByMap = SpaceCurveOptimizeHelper.createOptimizedDataFrameByMapValue(df, "c1_int, c2_int", 200, "hilbert")
+ val dfHilbertOptimizeBySample = SpaceCurveOptimizeHelper.createOptimizeDataFrameBySample(df, "c1_int, c2_int", 200, "hilbert")
+
+ saveAsTable(dfOptimizeByMap, tablePath, "z_sort_byMap")
+ saveAsTable(dfOptimizeBySample, tablePath, "z_sort_bySample")
+ saveAsTable(dfHilbertOptimizeByMap, tablePath, "hilbert_sort_byMap")
+ saveAsTable(dfHilbertOptimizeBySample, tablePath, "hilbert_sort_bySample")
+ }
+
+ def prepareSkewTable(tablePath: Path, numRows: Int): Unit = {
Review comment:
No need to duplicate this, instead please parameterize the other method to either accept desired range of the random point axis, or generator producing random point as a whole
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978978256
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45e4d2175d249bc68889b807a1833ac6fac434c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739)
* 4bd2f9aeab89bdd209db4d64b1f5a95696717c5d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744)
* d05366ef0fab06fab3a6ec283ab6b12069fbaba4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r754825971
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestOptimizeTable.scala
##########
@@ -64,8 +64,11 @@ class TestOptimizeTable extends HoodieClientTestBase {
}
@ParameterizedTest
- @ValueSource(strings = Array("COPY_ON_WRITE", "MERGE_ON_READ"))
- def testOptimizewithClustering(tableType: String): Unit = {
+ @ValueSource(strings = Array("COPY_ON_WRITE, hilbert", "COPY_ON_WRITE, z-order", "MERGE_ON_READ, hilbert", "MERGE_ON_READ, z-order"))
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r755639715
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
+ X[2] = C F I L O axes |/
+ high low 0------> X[0]
+ * </pre>
+ *
+ * @param index
+ * index to be tranposed
+ * @return transposed index
+ */
+ long[] transpose(BigInteger index) {
+ long[] x = new long[dimensions];
+ transpose(index, x);
+ return x;
+ }
+
+ private void transpose(BigInteger index, long[] x) {
+ byte[] b = index.toByteArray();
+ for (int idx = 0; idx < 8 * b.length; idx++) {
+ if ((b[b.length - 1 - idx / 8] & (1L << (idx % 8))) != 0) {
+ int dim = (length - idx - 1) % dimensions;
+ int shift = (idx / dimensions) % bits;
+ x[dim] |= 1L << shift;
+ }
+ }
+ }
+
+ /**
+ * <p>
+ * Given the axes (coordinates) of a point in N-Dimensional space, find the
+ * distance to that point along the Hilbert curve. That distance will be
+ * transposed; broken into pieces and distributed into an array.
+ *
+ * <p>
+ * The number of dimensions is the length of the hilbertAxes array.
+ *
+ * <p>
+ * Note: In Skilling's paper, this function is called AxestoTranspose.
+ *
+ * @param bits
+ * @param point
+ * Point in N-space
+ * @return The Hilbert distance (or index) as a transposed Hilbert index
+ */
+ static long[] transposedIndex(int bits, long... point) {
+ final long M = 1L << (bits - 1);
+ final int n = point.length; // n: Number of dimensions
+ final long[] x = Arrays.copyOf(point, n);
+ long p;
+ long q;
+ long t;
Review comment:
agree
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
+ X[2] = C F I L O axes |/
+ high low 0------> X[0]
+ * </pre>
+ *
+ * @param index
+ * index to be tranposed
+ * @return transposed index
+ */
+ long[] transpose(BigInteger index) {
+ long[] x = new long[dimensions];
+ transpose(index, x);
+ return x;
+ }
+
+ private void transpose(BigInteger index, long[] x) {
+ byte[] b = index.toByteArray();
+ for (int idx = 0; idx < 8 * b.length; idx++) {
+ if ((b[b.length - 1 - idx / 8] & (1L << (idx % 8))) != 0) {
+ int dim = (length - idx - 1) % dimensions;
+ int shift = (idx / dimensions) % bits;
+ x[dim] |= 1L << shift;
+ }
+ }
+ }
+
+ /**
+ * <p>
+ * Given the axes (coordinates) of a point in N-Dimensional space, find the
+ * distance to that point along the Hilbert curve. That distance will be
+ * transposed; broken into pieces and distributed into an array.
+ *
+ * <p>
+ * The number of dimensions is the length of the hilbertAxes array.
+ *
+ * <p>
+ * Note: In Skilling's paper, this function is called AxestoTranspose.
+ *
+ * @param bits
+ * @param point
+ * Point in N-space
+ * @return The Hilbert distance (or index) as a transposed Hilbert index
+ */
+ static long[] transposedIndex(int bits, long... point) {
Review comment:
ok, will add UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r756614458
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
+ X[2] = C F I L O axes |/
+ high low 0------> X[0]
+ * </pre>
+ *
+ * @param index
+ * index to be tranposed
+ * @return transposed index
+ */
+ long[] transpose(BigInteger index) {
+ long[] x = new long[dimensions];
+ transpose(index, x);
+ return x;
+ }
+
+ private void transpose(BigInteger index, long[] x) {
+ byte[] b = index.toByteArray();
+ for (int idx = 0; idx < 8 * b.length; idx++) {
+ if ((b[b.length - 1 - idx / 8] & (1L << (idx % 8))) != 0) {
+ int dim = (length - idx - 1) % dimensions;
+ int shift = (idx / dimensions) % bits;
+ x[dim] |= 1L << shift;
+ }
+ }
+ }
+
+ /**
+ * <p>
+ * Given the axes (coordinates) of a point in N-Dimensional space, find the
+ * distance to that point along the Hilbert curve. That distance will be
+ * transposed; broken into pieces and distributed into an array.
+ *
+ * <p>
+ * The number of dimensions is the length of the hilbertAxes array.
+ *
+ * <p>
+ * Note: In Skilling's paper, this function is called AxestoTranspose.
+ *
+ * @param bits
+ * @param point
+ * Point in N-space
+ * @return The Hilbert distance (or index) as a transposed Hilbert index
+ */
+ static long[] transposedIndex(int bits, long... point) {
+ final long M = 1L << (bits - 1);
+ final int n = point.length; // n: Number of dimensions
+ final long[] x = Arrays.copyOf(point, n);
+ long p;
+ long q;
+ long t;
Review comment:
i try iy， but failed with code style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978897800
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a2c78442579ffd076f752f4b65d41778e482549 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732)
* 45e4d2175d249bc68889b807a1833ac6fac434c5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978899543
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a2c78442579ffd076f752f4b65d41778e482549 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732)
* 45e4d2175d249bc68889b807a1833ac6fac434c5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966119536
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a3305e70773578729177c6ee863d52ecf31ee39 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235)
* 60c291565f976aa39d5e94591ab58079230c2358 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-979084670
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d05366ef0fab06fab3a6ec283ab6b12069fbaba4 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747)
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980451466
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824)
* 0d034fb15666134c98cfff743715ec0aeee29602 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980456259
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824)
* 0d034fb15666134c98cfff743715ec0aeee29602 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980499225
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r757695171
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980481836
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840)
* 4d1777324ea0588cf68ee56c439610278e222192 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980490078
@yihua could you pls help me，now i cannot pull/push any code in github。
those are my modify：
patch1： modify HilbertCurveUitls
public class HilbertCurveUtils {
public static byte[] indexBytes(HilbertCurve hilbertCurve, long[] points, int paddingNum) {
BigInteger index = hilbertCurve.index(points);
return paddingToNByte(index.toByteArray(), paddingNum);
}
public static byte[] paddingToNByte(byte[] a, int paddingNum) {
if (a.length == paddingNum) {
return a;
}
if (a.length > paddingNum) {
byte[] result = new byte[paddingNum];
System.arraycopy(a, 0, result, 0, paddingNum);
return result;
}
int paddingSize = 8 - a.length;
byte[] result = new byte[paddingNum];
for (int i = 0; i < paddingSize; i++) {
result[i] = 0;
}
System.arraycopy(a, 0, result, paddingSize, a.length);
return result;
}
}
patch2：
line220 in OrderingIndexHelper:
hilbertCurve, longList.stream().mapToLong(l -> l).toArray(), 63);
patch3:
line520 in RangeSample:
HilbertCurveUtils.indexBytes(hilbertCurve.get, values.map(_.toLong).toArray, 32)
patch4:
line 99 in SpaceCurveOptimizeBenchMark:
val df = spark.range(numRows).map(_ => (Random.nextInt(col1Range), Random.nextInt(col2Range))).toDF("c1_int", "c2_int")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980488165
@xiarixiaoyao do you see if the above is the root cause of the poor performance? If not, let's revert to your original code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980504819
@xiarixiaoyao @vinothchandar I try to run clustering with Hilbert Curve in Spark Shell using Spark datasource, but from Spark UI, Z-Ordering is still being used. It seems to be off.
<img width="1722" alt="Screen Shot 2021-11-26 at 21 11 34" src="https://user-images.githubusercontent.com/2497195/143668968-11b7f139-f0ba-493c-827a-5086c45b8390.png">
```
import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
val tableName = "hudi_trips_cow"
val basePath = "file:///tmp/hudi_trips_cow"
val dataGen = new DataGenerator
val inserts = convertToStringList(dataGen.generateInserts(1000))
val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
df.write.format("hudi").
option("hoodie.insert.shuffle.parallelism", "2").
option("hoodie.upsert.shuffle.parallelism", "2").
option("hoodie.bulkinsert.shuffle.parallelism", "2").
option("hoodie.delete.shuffle.parallelism", "2").
option(PRECOMBINE_FIELD_OPT_KEY, "ts").
option(RECORDKEY_FIELD_OPT_KEY, "uuid").
option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
option(TABLE_NAME, tableName).
option("hoodie.parquet.small.file.limit", "0").
option("hoodie.clustering.inline", "true").
option("hoodie.clustering.inline.max.commits", "2").
option("hoodie.clustering.plan.strategy.target.file.max.bytes", "1073741824").
option("hoodie.clustering.plan.strategy.small.file.limit", "629145600").
option("hoodie.clustering.plan.strategy.sort.columns", "rider,driver").
option("hoodie.layout.optimize.enable", "true").
option("hoodie.layout.optimize.strategy", "hilbert").
mode(Append).
save(basePath)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980493766
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 4d1777324ea0588cf68ee56c439610278e222192 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842)
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980491476
@xiarixiaoyao I pushed a commit based on your patch. Let me know if that looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980448546
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980474439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980481582
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840)
* 4d1777324ea0588cf68ee56c439610278e222192 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102][WIP] support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-963871181
@vinothchandar
Hilbert curve has better data aggregation characteristics, while Z curve has worse data aggregation characteristics.
The mapping process of Hilbert curve is complex, and the mapping process of Z curve is simple.
High order Hilbert computation is very complex and time-consuming, and z-order is used more in spatio-temporal database.
i will paste perf test later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966212378
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r754782757
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
Review comment:
@alexeykudinkin can you please expand. so @xiarixiaoyao can address/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r754806140
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java
##########
@@ -458,4 +458,27 @@ public static BuildCurveStrategyType fromValue(String value) {
}
}
}
+
+ /**
+ * strategy types for optimize layout for hudi data.
+ */
+ public enum BuildLayoutOptimizationStrategy {
+ ZORDER("z-order"),
+ HILBERT("hilbert");
+ private final String value;
+ BuildLayoutOptimizationStrategy(String value) {
+ this.value = value;
+ }
+
+ public static BuildLayoutOptimizationStrategy fromValue(String value) {
Review comment:
There's default `valueOf` in every Enum. What do we need this method for?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
Review comment:
Let's pass an array instead of var-args (var-arg is a better fit for loose-form method args, rather than well-scoped N-dimensional point)
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
Review comment:
Do we need this extra linebreak here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r754826477
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
Review comment:
agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r754826413
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
Review comment:
ok , let me fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978897800
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a2c78442579ffd076f752f4b65d41778e482549 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732)
* 45e4d2175d249bc68889b807a1833ac6fac434c5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978897162
@vinothchandar @alexeykudinkin
addressed all comments;
update the code
1) add microBenchmark code for z-order/hilbert and test result.
2) add more UTS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978955941
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45e4d2175d249bc68889b807a1833ac6fac434c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739)
* 4bd2f9aeab89bdd209db4d64b1f5a95696717c5d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978978256
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45e4d2175d249bc68889b807a1833ac6fac434c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739)
* 4bd2f9aeab89bdd209db4d64b1f5a95696717c5d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744)
* d05366ef0fab06fab3a6ec283ab6b12069fbaba4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980456259
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824)
* 0d034fb15666134c98cfff743715ec0aeee29602 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980473749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d034fb15666134c98cfff743715ec0aeee29602 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830)
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r757722700
##########
File path: hudi-client/hudi-client-common/src/test/java/org/apache/hudi/optimize/TestZOrderingUtil.java
##########
@@ -126,4 +126,21 @@ public OrginValueWrapper(T index, T originValue) {
this.originValue = originValue;
}
}
+
+ @Test
+ public void testConvertBytesToLong() {
Review comment:
Added one more test for the padding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980481582
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840)
* 4d1777324ea0588cf68ee56c439610278e222192 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r757730126
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/spark/OrderingIndexHelper.java
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieColumnRangeMetadata;
+import org.apache.hudi.common.model.HoodieFileFormat;
+import org.apache.hudi.common.util.BaseFileUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.ParquetUtils;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.index.zorder.ZOrderingIndexHelper;
+import org.apache.hudi.optimize.HilbertCurveUtils;
+import org.apache.hudi.optimize.ZOrderingUtil;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.io.api.Binary;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.Row$;
+import org.apache.spark.sql.SparkSession;
+import org.apache.spark.sql.hudi.execution.RangeSampleSort$;
+import org.apache.spark.sql.hudi.execution.ZorderingBinarySort;
+import org.apache.spark.sql.types.BinaryType;
+import org.apache.spark.sql.types.BinaryType$;
+import org.apache.spark.sql.types.BooleanType;
+import org.apache.spark.sql.types.ByteType;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.DateType;
+import org.apache.spark.sql.types.DecimalType;
+import org.apache.spark.sql.types.DoubleType;
+import org.apache.spark.sql.types.FloatType;
+import org.apache.spark.sql.types.IntegerType;
+import org.apache.spark.sql.types.LongType;
+import org.apache.spark.sql.types.LongType$;
+import org.apache.spark.sql.types.Metadata;
+import org.apache.spark.sql.types.ShortType;
+import org.apache.spark.sql.types.StringType;
+import org.apache.spark.sql.types.StringType$;
+import org.apache.spark.sql.types.StructField;
+import org.apache.spark.sql.types.StructType$;
+import org.apache.spark.sql.types.TimestampType;
+import org.apache.spark.util.SerializableConfiguration;
+import org.davidmoten.hilbert.HilbertCurve;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import scala.collection.JavaConversions;
+
+public class OrderingIndexHelper {
Review comment:
sorry, it should be in org.apache.hudi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao edited a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao edited a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980486706
for table table_z_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.795
for table table_z_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent 0.76
for table table_hilbert_sort_byMap with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent **0.18999999999999995**
for table table_hilbert_sort_bySample with query filter: c1_int = 500000 or c2_int = 500000 we can achieve skipping percent **0.21999999999999997**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980491542
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840)
* 4d1777324ea0588cf68ee56c439610278e222192 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842)
* 503ebc7ff871791004511bae1bc79465e341062d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980498548
@xiarixiaoyao Some tests in `TestTableLayoutOptimization` fail. Could you help fix those?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966178942
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966178942
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978954164
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45e4d2175d249bc68889b807a1833ac6fac434c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978899543
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a2c78442579ffd076f752f4b65d41778e482549 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732)
* 45e4d2175d249bc68889b807a1833ac6fac434c5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978867417
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8a2c78442579ffd076f752f4b65d41778e482549 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r756621064
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieClusteringConfig.java
##########
@@ -458,4 +458,27 @@ public static BuildCurveStrategyType fromValue(String value) {
}
}
}
+
+ /**
+ * strategy types for optimize layout for hudi data.
+ */
+ public enum BuildLayoutOptimizationStrategy {
+ ZORDER("z-order"),
+ HILBERT("hilbert");
+ private final String value;
+ BuildLayoutOptimizationStrategy(String value) {
+ this.value = value;
+ }
+
+ public static BuildLayoutOptimizationStrategy fromValue(String value) {
Review comment:
Why can't it use default `valueOf`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980480799
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839)
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980481068
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839)
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980479531
@yihua Sorry, due to the time difference, I only see these comments now. Thank you very much for your help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980483111
@xiarixiaoyao @vinothchandar the bundle dependency issue is resolved. This PR is ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980486636
@yihua @vinothchandar
I found a serious performance problem that we can't use directly
com.github.davidmoten: the Hilbert curve package has a very bad performance degradation after using it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980488386
@yihua i think i find the root cause. let me updated the code. no need to revert to original code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980490046
> @vinothchandar @xiarixiaoyao We already updated NOTICE. So we can revert back to that. I am debugging some issue with z-ordering on a larger dataset.
>
> what's the problems. may be i can give some help.
@vinothchandar pinged you on the Slack and added you to a channel. Let us know if you have seen that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980502594
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980505992
> @yihua This is about the logic of saving statistics, which is normal. hilbert and zorder will use the same logical to save statistic info
Oh, got it. It makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966178151
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-966175269
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r748668692
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
Review comment:
This seems off
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3952: [HUDI-2102][WIP] support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-963783688
IIUC hilbert curves are actually more prevalent in database systems. curious to see how it compares to z-order curves!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102][WIP] support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-963806283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4a3305e70773578729177c6ee863d52ecf31ee39 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r755854131
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
+ X[2] = C F I L O axes |/
+ high low 0------> X[0]
+ * </pre>
+ *
+ * @param index
+ * index to be tranposed
+ * @return transposed index
+ */
+ long[] transpose(BigInteger index) {
+ long[] x = new long[dimensions];
+ transpose(index, x);
+ return x;
+ }
+
+ private void transpose(BigInteger index, long[] x) {
+ byte[] b = index.toByteArray();
+ for (int idx = 0; idx < 8 * b.length; idx++) {
+ if ((b[b.length - 1 - idx / 8] & (1L << (idx % 8))) != 0) {
+ int dim = (length - idx - 1) % dimensions;
+ int shift = (idx / dimensions) % bits;
+ x[dim] |= 1L << shift;
+ }
+ }
+ }
+
+ /**
+ * <p>
+ * Given the axes (coordinates) of a point in N-Dimensional space, find the
+ * distance to that point along the Hilbert curve. That distance will be
+ * transposed; broken into pieces and distributed into an array.
+ *
+ * <p>
+ * The number of dimensions is the length of the hilbertAxes array.
+ *
+ * <p>
+ * Note: In Skilling's paper, this function is called AxestoTranspose.
+ *
+ * @param bits
+ * @param point
+ * Point in N-space
Review comment:
These notes are from the original author. Sorry for not paying attention to these notes, let me fixed them。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r754805860
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
+ * the Licensed of above link is also http://www.apache.org/licenses/LICENSE-2.0
+ */
+public final class HilbertCurve {
+
+ private final int bits;
+ private final int dimensions;
+ // cached calculations
+ private final int length;
+
+ private HilbertCurve(int bits, int dimensions) {
+ this.bits = bits;
+ this.dimensions = dimensions;
+ // cache a calculated values for small perf improvements
+ this.length = bits * dimensions;
+ }
+
+ /**
+ * Returns a builder for and object that performs transformations for a Hilbert
+ * curve with the given number of bits.
+ *
+ * @param bits
+ * depth of the Hilbert curve. If bits is one, this is the top-level
+ * Hilbert curve
+ * @return builder for object to do transformations with the Hilbert Curve
+ */
+ public static Builder bits(int bits) {
+ return new Builder(bits);
+ }
+
+ /**
+ * Builds a {@link HilbertCurve} instance.
+ */
+ public static final class Builder {
+ final int bits;
+
+ private Builder(int bits) {
+ if (bits <= 0 || bits >= 64) {
+ throw new IllegalArgumentException(String.format("bits must be greater than zero and less than 64, now found bits value: %s", bits));
+ }
+ this.bits = bits;
+ }
+
+ public HilbertCurve dimensions(int dimensions) {
+ if (dimensions < 2) {
+ throw new IllegalArgumentException(String.format("dimensions must be at least 2, now found dimensions value: %s", dimensions));
+ }
+ return new HilbertCurve(bits, dimensions);
+ }
+ }
+
+ /**
+ * Converts a point to its Hilbert curve index.
+ *
+ * @param point
+ * an array of {@code long}. Each ordinate can be between 0 and
+ * 2<sup>bits</sup>-1.
+ * @return index (nonnegative {@link BigInteger})
+ * @throws IllegalArgumentException
+ * if length of point array is not equal to the number of
+ * dimensions.
+ */
+ public BigInteger index(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndex(transposedIndex(bits, point));
+ }
+
+ public byte[] indexBytes(long... point) {
+ if (point.length != dimensions) {
+ throw new IllegalArgumentException(String.format("length of point array must equal to the number of dimensions"));
+ }
+ return toIndexBytes(transposedIndex(bits, point));
+ }
+
+ /**
+ * Converts a {@link BigInteger} index (distance along the Hilbert Curve from 0)
+ * to a point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2 <sup>bits *
+ * dimensions</sup>-1.
+ * @return array of longs being the point
+ * @throws NullPointerException
+ * if index is null
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(BigInteger index) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ return transposedIndexToPoint(bits, transpose(index));
+ }
+
+ public void point(BigInteger index, long[] x) {
+ if (index == null) {
+ throw new NullPointerException("index must not be null");
+ }
+ if (index.signum() == -1) {
+ throw new IllegalArgumentException("index cannot be negative");
+ }
+ Arrays.fill(x, 0);
+ transpose(index, x);
+ transposedIndexToPoint(bits, x);
+ }
+
+ public void point(long i, long[] x) {
+ point(BigInteger.valueOf(i), x);
+ }
+
+ /**
+ * Converts a {@code long} index (distance along the Hilbert Curve from 0) to a
+ * point of dimensions defined in the constructor of {@code this}.
+ *
+ * @param index
+ * index along the Hilbert Curve from 0. Maximum value 2
+ * <sup>bits+1</sup>-1.
+ * @return array of longs being the point
+ * @throws IllegalArgumentException
+ * if index is negative
+ */
+ public long[] point(long index) {
+ return point(BigInteger.valueOf(index));
+ }
+
+ /**
+ * Returns the transposed representation of the Hilbert curve index.
+ *
+ * <p>
+ * The Hilbert index is expressed internally as an array of transposed bits.
+ *
+ * <pre>
+ Example: 5 bits for each of n=3 coordinates.
+ 15-bit Hilbert integer = A B C D E F G H I J K L M N O is stored
+ as its Transpose ^
+ X[0] = A D G J M X[2]| 7
+ X[1] = B E H K N <-------> | /X[1]
Review comment:
Sorry, for not being elaborate enough -- i was referring to escaped symbols `<------->`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r754814934
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/HilbertCurve.java
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.optimize;
+
+import java.math.BigInteger;
+import java.util.Arrays;
+
+/**
+ * Converts between Hilbert index ({@code BigInteger}) and N-dimensional points.
+ *
+ * <p>
+ * Note:
+ * <a href="https://github.com/davidmoten/hilbert-curve/blob/master/src/main/java/org/davidmoten/hilbert/HilbertCurve.java">GitHub</a>).
Review comment:
yes, it should be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978836367
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 60c291565f976aa39d5e94591ab58079230c2358 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307)
* 8a2c78442579ffd076f752f4b65d41778e482549 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r756647750
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestTableLayoutOptimization.scala
##########
@@ -68,8 +66,8 @@ class TestTableLayoutOptimization extends HoodieClientTestBase {
}
@ParameterizedTest
- @ValueSource(strings = Array("COPY_ON_WRITE", "MERGE_ON_READ"))
- def testOptimizeWithClustering(tableType: String): Unit = {
+ @CsvSource(Array("COPY_ON_WRITE, hilbert", "COPY_ON_WRITE, z-order", "MERGE_ON_READ, hilbert", "MERGE_ON_READ, z-order"))
Review comment:
@alexeykudinkin could you pls help me， i donnot know how to use @MethodSource in scala。 thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978981419
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4bd2f9aeab89bdd209db4d64b1f5a95696717c5d Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744)
* d05366ef0fab06fab3a6ec283ab6b12069fbaba4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-978958533
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 45e4d2175d249bc68889b807a1833ac6fac434c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739)
* 4bd2f9aeab89bdd209db4d64b1f5a95696717c5d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980442192
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1c91d564bcb248a58de5b408f4eb8d1f91823c7e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749)
* cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#discussion_r757695928
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/optimize/ZOrderingUtil.java
##########
@@ -176,9 +176,17 @@ public static byte updatePos(byte a, int apos, byte b, int bpos) {
public static Long convertStringToLong(String a) {
byte[] bytes = utf8To8Byte(a);
+ return convertBytesToLong(bytes);
+ }
+
+ public static long convertBytesToLong(byte[] bytes) {
+ byte[] padBytes = bytes;
Review comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980480799
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 411029ac026d4a7fa528a486240837d775fe3a75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839)
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980483169
@yihua pls wait, after some fixed the benchmark cannot run let me check it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980483457
> @yihua pls wait, after some fixed the benchmark cannot run let me check it
Got it. I let you check that. Let me know when it's good to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980505736
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852",
"triggerID" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850)
* 18d2357845c09748e5cd7297b3d3f4569baad7d8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3852)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980502905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850",
"triggerID" : "980502594",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3850)
* 18d2357845c09748e5cd7297b3d3f4569baad7d8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980500631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
}, {
"hash" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "18d2357845c09748e5cd7297b3d3f4569baad7d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* 503ebc7ff871791004511bae1bc79465e341062d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3844)
* 18d2357845c09748e5cd7297b3d3f4569baad7d8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980491542
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3235",
"triggerID" : "4a3305e70773578729177c6ee863d52ecf31ee39",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3305",
"triggerID" : "60c291565f976aa39d5e94591ab58079230c2358",
"triggerType" : "PUSH"
}, {
"hash" : "60c291565f976aa39d5e94591ab58079230c2358",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3307",
"triggerID" : "966178151",
"triggerType" : "MANUAL"
}, {
"hash" : "8a2c78442579ffd076f752f4b65d41778e482549",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3732",
"triggerID" : "8a2c78442579ffd076f752f4b65d41778e482549",
"triggerType" : "PUSH"
}, {
"hash" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3739",
"triggerID" : "45e4d2175d249bc68889b807a1833ac6fac434c5",
"triggerType" : "PUSH"
}, {
"hash" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3744",
"triggerID" : "4bd2f9aeab89bdd209db4d64b1f5a95696717c5d",
"triggerType" : "PUSH"
}, {
"hash" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3747",
"triggerID" : "d05366ef0fab06fab3a6ec283ab6b12069fbaba4",
"triggerType" : "PUSH"
}, {
"hash" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3749",
"triggerID" : "1c91d564bcb248a58de5b408f4eb8d1f91823c7e",
"triggerType" : "PUSH"
}, {
"hash" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3824",
"triggerID" : "cb2fc3ecc6082e82b2bf4b2abd636ad8cdfc04b0",
"triggerType" : "PUSH"
}, {
"hash" : "0d034fb15666134c98cfff743715ec0aeee29602",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3830",
"triggerID" : "0d034fb15666134c98cfff743715ec0aeee29602",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3836",
"triggerID" : "411029ac026d4a7fa528a486240837d775fe3a75",
"triggerType" : "PUSH"
}, {
"hash" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "688052e6a4e22f30bd6728609edea968d9cc1bfd",
"triggerType" : "PUSH"
}, {
"hash" : "411029ac026d4a7fa528a486240837d775fe3a75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3839",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"triggerType" : "PUSH"
}, {
"hash" : "da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840",
"triggerID" : "980480518",
"triggerType" : "MANUAL"
}, {
"hash" : "4d1777324ea0588cf68ee56c439610278e222192",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842",
"triggerID" : "4d1777324ea0588cf68ee56c439610278e222192",
"triggerType" : "PUSH"
}, {
"hash" : "503ebc7ff871791004511bae1bc79465e341062d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "503ebc7ff871791004511bae1bc79465e341062d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 688052e6a4e22f30bd6728609edea968d9cc1bfd UNKNOWN
* da9b8fc0a3230a3b4233f4c1d5a4bec8399cfc4f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3840)
* 4d1777324ea0588cf68ee56c439610278e222192 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3842)
* 503ebc7ff871791004511bae1bc79465e341062d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #3952: [HUDI-2102]support hilbert curve for hudi.
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on pull request #3952:
URL: https://github.com/apache/hudi/pull/3952#issuecomment-980488803
@vinothchandar
@xiarixiaoyao We already updated NOTICE. So we can revert back to that.
I am debugging some issue with z-ordering on a larger dataset.
what's the problems. may be i can give some help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org