You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "kazdy (via GitHub)" <gi...@apache.org> on 2023/05/16 11:13:34 UTC
[GitHub] [hudi] kazdy opened a new pull request, #8729: [DRAFT] Support Incremental read via Spark SQL table valued function
kazdy opened a new pull request, #8729:
URL: https://github.com/apache/hudi/pull/8729
### Change Logs
Support incremental read (both latest_state and cdc) for Spark SQL via table valued function.
### Impact
New hudi_table_changes() function is added to Hudi Spark SQL Extension
### Risk level (write none, low medium or high below)
Low
### Documentation Update
add documentation for hudi_table_changes() function:
- hudi_table_changes('identifier', 'latest_state | cdc', 'earliest') -> start from earliest available commit.
- hudi_table_changes('identifier', 'latest_state | cdc', 'earliest', '202305160000') -> start from earliest, end at 202305160000.
- hudi_table_changes('identifier', 'latest_state | cdc', '202305150000', '202305160000') -> start from 202305150000, end at 202305160000.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1596721239
@kazdy
I have reviewed and attached a patch here, the basic idea is to merge these two functions into one, the first param would become polymophic.
You can apply the patch first then rebase with master code to resolve conflicts:
[6223.patch.zip](https://github.com/apache/hudi/files/11785842/6223.patch.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1565762535
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13ec59e5c290d52c92cb9590a1cadb43eface974 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1596990277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
}, {
"hash" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588",
"triggerID" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"triggerType" : "PUSH"
}, {
"hash" : "a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b846e14a7d153415da9392c407019cbc5ce7a5a5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588)
* a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1217792275
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
> This does not feel like the right thing to do.
I wanted it to be as easy to use as it can be, just pass arguments
The SQL standard in 2017 defines that the table valued function could take named param, it deserves a try here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1218690154
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
Ok i found this https://www.postgresql.org/docs/current/sql-syntax-calling-funcs.html#SQL-SYNTAX-CALLING-FUNCS-NAMED
Still similar syntax is not supported in Spark SQL. I think it will be nice to use it, but only after there's support for it in spark.
Wdyt Danny?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #8729:
URL: https://github.com/apache/hudi/pull/8729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jonvex commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1564724152
Could you please update `hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlCoreFlow.scala` lines 118-142 to use this instead of spark ds incremental query? Let me know if I can help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1593981495
Hi, @kazdy , are there any updates here, the 0.14.0 release code freeze is due to the next weekend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1575044294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6451d26be69f5d61b58c08539f0e72b5f99e64cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585)
* 54d27f1f45cb117f785a1a2bfcbf0562358e2e13 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1217660242
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
AFAIK TVF in Spark unlike procedures doesn't support named arguments natively.
Looking at Spark docs: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-tvf.html there are none that have named arguments.
So to handle everything in one TVF it would look like this:
```
hudi_table_changes('path', 'my/table', 'latest_state', '12345678')
hudi_table_changes('table', 'myTable', 'cdc', '12345678')
```
Or I would need to work against what Spark offers and parse named arguments myself to have:
```
hudi_table_changes('path=my/table', 'type=latest_state', 'beginInstantTime=12345678')
hudi_table_changes('table=myTable', 'type=cdc', 'beginInstantTime=12345678')
```
This does not feel like the right thing to do.
I wanted it to be as easy to use as it can be, just pass arguments.
In the case of path and table I can't assume what will be passed by the user and it would be hard to infer this from the input, but latest_state or cdc I can so I could do it easily and introduce this flag.
To be honest I was thinking in the other direction, to split cdc and latest_state to separate functions:
```
hudi_table_changes('table', 'earliest', '123456789')
hudi_table_changes_by_path('my/table', 'earliest', '123456789')
hudi_table_cdc_changes('table', 'earliest', '123456789')
hudi_table_cdc_changes_by_path('my/table', 'earliest', '123456789')
```
But the version I ended up with seemed to be a good compromise so I went with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1218794472
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
[c069776_ISO_IEC_TR_19075-7_2017.pdf](https://github.com/apache/hudi/files/11659369/c069776_ISO_IEC_TR_19075-7_2017.pdf)
Here is the part about table valued functions from SQL standard 2017, is there any possibility we can add a custom syntax parse block for Hudi then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1549828573
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124)
* 7578c91f0029a9eb45119f33bf4decedbb3c6939 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1565697857
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1612fdfe0433cf8f62d5834490efac6730ef6209 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422)
* 13ec59e5c290d52c92cb9590a1cadb43eface974 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1565694968
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1612fdfe0433cf8f62d5834490efac6730ef6209 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422)
* 13ec59e5c290d52c92cb9590a1cadb43eface974 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1575034816
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6451d26be69f5d61b58c08539f0e72b5f99e64cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1575040215
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6451d26be69f5d61b58c08539f0e72b5f99e64cd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585)
* 54d27f1f45cb117f785a1a2bfcbf0562358e2e13 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1217387626
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
Glance over the changes quickly and have some concerns:
1. where does the name like `hudi_table_changes` and `hudi_table_changes_by_path` come from, like they both work for incremental queries
2. if we can impl the named param for the table value functions, we can merge the two functions into one, for named param, it looks like: `table=xxx, ...` or `path=xxx, ...`, and we can even enable/disable the cdc loggins with the param
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1219081012
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
Thanks! I'll check and give it a try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1574933938
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13ec59e5c290d52c92cb9590a1cadb43eface974 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443)
* 6451d26be69f5d61b58c08539f0e72b5f99e64cd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1217660242
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
AFAIK TVF in Spark unlike procedures doesn't support named arguments natively.
Looking at Spark docs: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-tvf.html there are none that have named arguments.
So to handle everything in one TVF it would look like this:
```
hudi_table_changes('path', 'my/table', 'latest_state', '12345678')
hudi_table_changes('table', 'myTable', 'cdc', '12345678')
```
Or I would need to work against what Spark offers and parse named arguments myself to have:
```
hudi_table_changes('path=my/table', 'type=latest_state', 'beginInstantTime=12345678')
hudi_table_changes('table=myTable', 'type=cdc', 'beginInstantTime=12345678')
```
This does not feel like the right thing to do.
I wanted it to be as easy to use as it can be, just pass arguments.
In the case of path and table I can't assume what will be passed by the user and it would be hard to infer this from the input, but latest_state or cdc I can so I could do it easily and introduce this flag.
To be honest I was thinking in the other direction, to split cdc and latest_state to separate functions:
```
hudi_table_changes('table', 'earliest', '123456789')
hudi_table_changes_by_path('my/table', 'earliest', '123456789')
hudi_table_cdc_changes('table', 'earliest', '123456789')
hudi_table_cdc_changes_by_path('my/table', 'earliest', '123456789')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1217898440
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
I only found that in one case in pgsql dblink function uses
```sql
select * from
dblink('dbname=mydb options=-csearch_path=', 'select proname, prosrc from pg_proc')
```
But that's a db connection string passed as one argument so not exactly the same case.
I've only seen tvf arguments to be resolved by position. The declaration of func arguments can be done by name, but I have not seen anywhere that I can call my function by using `select * from myfunc(path='abc')`.
I like this:
```
hudi_table_changes(path='xxx/yyy', 'cdc', 123456, 123456)
```
but this not so much:
```
hudi_table_changes('path=xxx/yyy', 'cdc', 123456, 123456)
```
and 1st is not supported in Spark SQL syntax atm.
I found one ticket that aims to implement named argument support in parser for function call:
https://issues.apache.org/jira/browse/SPARK-43922
targeted for Spark 3.5, so maybe this can be done in the future in an elegant way.
This is how another implementation for a similar function in spark was done:
https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed#sql
Nevertheless, what should the syntax in the mentioned standard look like then? I was not able to find any information on what the sql standard defines.
Can you give me an example? Should only the first argument be either "path=xxx/xxx", "identifier=db.tableName", or "identifier=tableName" and the rest stay as is so that users don't need to write too much in their queries?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1550200400
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b35c8ce62728cba8c2760272bce8e9171e1eea31 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130)
* ab89b324fde088312ac07e50425462b67b54393b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1217660242
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
AFAIK TVF in Spark unlike procedures doesn't support named arguments natively.
Looking at Spark docs: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-tvf.html there are none that have named arguments.
So to handle everything in one TVF it would look like this:
```
hudi_table_changes('path', 'my/table', 'latest_state', '12345678')
hudi_table_changes('table', 'myTable', 'cdc', '12345678')
```
Or I would need to work against what Spark offers and parse named arguments myself to have:
```
hudi_table_changes('path=my/table', 'type=latest_state', 'beginInstantTime=12345678')
hudi_table_changes('table=myTable', 'type=cdc', 'beginInstantTime=12345678')
```
This does not feel like the right thing to do.
I wanted it to be as easy to use as it can be, just pass arguments, and keep it consistent with what Spark SQL users are used to.
In the case of path and table I can't assume what will be passed by the user and it would be hard to infer this from the input, but latest_state or cdc I can expect so I could do it easily and introduce this flag.
To be honest I was thinking in the other direction, to split cdc and latest_state to separate functions:
```
hudi_table_changes('table', 'earliest', '123456789')
hudi_table_changes_by_path('my/table', 'earliest', '123456789')
hudi_table_cdc_changes('table', 'earliest', '123456789')
hudi_table_cdc_changes_by_path('my/table', 'earliest', '123456789')
```
But the version I ended up with seemed to be a good compromise so I went with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1597075583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
}, {
"hash" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588",
"triggerID" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"triggerType" : "PUSH"
}, {
"hash" : "a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17975",
"triggerID" : "a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b",
"triggerType" : "PUSH"
}, {
"hash" : "855569b2998b85356d846ba07402cbb544b5623c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "855569b2998b85356d846ba07402cbb544b5623c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b846e14a7d153415da9392c407019cbc5ce7a5a5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588)
* a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17975)
* 855569b2998b85356d846ba07402cbb544b5623c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1549844519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124)
* 7578c91f0029a9eb45119f33bf4decedbb3c6939 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1565029309
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab89b324fde088312ac07e50425462b67b54393b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132)
* 1612fdfe0433cf8f62d5834490efac6730ef6209 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1575161853
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
}, {
"hash" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54d27f1f45cb117f785a1a2bfcbf0562358e2e13 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587)
* b846e14a7d153415da9392c407019cbc5ce7a5a5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1574939152
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 13ec59e5c290d52c92cb9590a1cadb43eface974 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443)
* 6451d26be69f5d61b58c08539f0e72b5f99e64cd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1598015265
@kazdy Can you fire another PR to add a document to the Hudi site about the usage of the new function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1597534685
> @kazdy
> I have reviewed and attached a patch here, the basic idea is to merge these two functions into one, the first param would become polymophic.
>
> You can apply the patch first then rebase with master code to resolve conflicts:
> [6223.patch.zip](https://github.com/apache/hudi/files/11785842/6223.patch.zip)
>
>
>
@danny0405 patched and rebased, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1575127619
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54d27f1f45cb117f785a1a2bfcbf0562358e2e13 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rahil-c commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1577485294
Thank you making this change @kazdy, are we targeting this for next hudi release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1217792275
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
> This does not feel like the right thing to do.
I wanted it to be as easy to use as it can be, just pass arguments
The SQL standard in 2017 definds that the table valued function could take named param, it deserves a try here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1549947789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7578c91f0029a9eb45119f33bf4decedbb3c6939 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129)
* b35c8ce62728cba8c2760272bce8e9171e1eea31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1550190944
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7578c91f0029a9eb45119f33bf4decedbb3c6939 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129)
* b35c8ce62728cba8c2760272bce8e9171e1eea31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130)
* ab89b324fde088312ac07e50425462b67b54393b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1549931885
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124)
* 7578c91f0029a9eb45119f33bf4decedbb3c6939 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129)
* b35c8ce62728cba8c2760272bce8e9171e1eea31 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1594237124
@danny0405 I was pretty busy lately and did not have time to extend spark sql syntax. So no updates unfortunately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1550529695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab89b324fde088312ac07e50425462b67b54393b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1596371628
@kazdy Can you rebase with the latest master to resolve the conflicts, we are about to code freeze and another solution is we land this first and explore other solutions in following releases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1549512738
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1565035302
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ab89b324fde088312ac07e50425462b67b54393b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132)
* 1612fdfe0433cf8f62d5834490efac6730ef6209 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1599162261
@danny0405 here's a PR with documentation #8880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1575222140
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
}, {
"hash" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588",
"triggerID" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b846e14a7d153415da9392c407019cbc5ce7a5a5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1217660242
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
AFAIK TVF in Spark unlike procedures doesn't support named arguments natively.
Looking at Spark docs: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-tvf.html there are none that have named arguments.
So to handle everything in one TVF it would look like this:
```
hudi_table_changes('path', 'my/table', 'latest_state', '12345678')
hudi_table_changes('table', 'myTable', 'cdc', '12345678')
```
Or I would need to work against what Spark offers and parse named arguments myself to have:
```
hudi_table_changes('path=my/table', 'type=latest_state', 'beginInstantTime=12345678')
hudi_table_changes('table=myTable', 'type=cdc', 'beginInstantTime=12345678')
```
This does not feel like the right thing to do.
I wanted it to be as easy to use as it can be, just pass arguments, and keep it consistent with what Spark SQL users are used to.
In the case of path and table I can't assume what will be passed by the user and it would be hard to infer this from the input, but latest_state or cdc I can so I could do it easily and introduce this flag.
To be honest I was thinking in the other direction, to split cdc and latest_state to separate functions:
```
hudi_table_changes('table', 'earliest', '123456789')
hudi_table_changes_by_path('my/table', 'earliest', '123456789')
hudi_table_cdc_changes('table', 'earliest', '123456789')
hudi_table_cdc_changes_by_path('my/table', 'earliest', '123456789')
```
But the version I ended up with seemed to be a good compromise so I went with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on a diff in pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #8729:
URL: https://github.com/apache/hudi/pull/8729#discussion_r1217898440
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logcal/HoodieTableChanges.scala:
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logcal
+
+import org.apache.hudi.common.util.ValidationUtils.checkState
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
+import org.apache.spark.sql.catalyst.plans.logical.LeafNode
+
+object HoodieTableChangesOptionsParser {
+ def parseOptions(exprs: Seq[Expression], funcName: String): (String, Map[String, String]) = {
+ val args = exprs.map(_.eval().toString)
+
+ if (args.size < 3) {
+ throw new AnalysisException(s"Too few arguments for function `$funcName`")
Review Comment:
I only found that in one case in pgsql dblink function uses
```sql
select * from
dblink('dbname=mydb options=-csearch_path=', 'select proname, prosrc from pg_proc')
```
But that's a db connection string passed as one argument so not exactly the same case.
I've only seen tvf arguments to be resolved by position. The declaration of func arguments can be done by name, but I have not seen anywhere that I can call my function by using `select * from myfunc(path='abc')`.
For spark I found one ticket that aims to implement named argument support in parser for function call:
https://issues.apache.org/jira/browse/SPARK-43922
targeted for Spark 3.5.
Tbh I'm not fully convinced if Spark SQL syntax does not support func call with named parameters.
This is how another implementation for similar function in spark:
https://learn.microsoft.com/en-us/azure/databricks/delta/delta-change-data-feed#sql
But what should the syntax in the mentioned standard look like then? I was not able to find any information on what the sql standard defines.
Can you give me an example? Should only the first argument be either "path=xxx/xxx", "identifier=db.tableName", or "identifier=tableName" and the rest stay as is so that users don't need to write too much in their queries?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1597087905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
}, {
"hash" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588",
"triggerID" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"triggerType" : "PUSH"
}, {
"hash" : "a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17975",
"triggerID" : "a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b",
"triggerType" : "PUSH"
}, {
"hash" : "855569b2998b85356d846ba07402cbb544b5623c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17977",
"triggerID" : "855569b2998b85356d846ba07402cbb544b5623c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17975)
* 855569b2998b85356d846ba07402cbb544b5623c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17977)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1597491033
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
}, {
"hash" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588",
"triggerID" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"triggerType" : "PUSH"
}, {
"hash" : "a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17975",
"triggerID" : "a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b",
"triggerType" : "PUSH"
}, {
"hash" : "855569b2998b85356d846ba07402cbb544b5623c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17977",
"triggerID" : "855569b2998b85356d846ba07402cbb544b5623c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 855569b2998b85356d846ba07402cbb544b5623c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17977)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1565134440
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1612fdfe0433cf8f62d5834490efac6730ef6209 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1575166193
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
}, {
"hash" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588",
"triggerID" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 54d27f1f45cb117f785a1a2bfcbf0562358e2e13 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587)
* b846e14a7d153415da9392c407019cbc5ce7a5a5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1575226203
I think this PR is in good enough shape now.
I would really appreciate a review @YannByron, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1574910025
> Could you please update `hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkSqlCoreFlow.scala` lines 118-142 to use this instead of spark ds incremental query? Let me know if I can help
done, there were no issues
thanks for mentioning this @jonvex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1597002845
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
}, {
"hash" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17129",
"triggerID" : "7578c91f0029a9eb45119f33bf4decedbb3c6939",
"triggerType" : "PUSH"
}, {
"hash" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17130",
"triggerID" : "b35c8ce62728cba8c2760272bce8e9171e1eea31",
"triggerType" : "PUSH"
}, {
"hash" : "ab89b324fde088312ac07e50425462b67b54393b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17132",
"triggerID" : "ab89b324fde088312ac07e50425462b67b54393b",
"triggerType" : "PUSH"
}, {
"hash" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17422",
"triggerID" : "1612fdfe0433cf8f62d5834490efac6730ef6209",
"triggerType" : "PUSH"
}, {
"hash" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17443",
"triggerID" : "13ec59e5c290d52c92cb9590a1cadb43eface974",
"triggerType" : "PUSH"
}, {
"hash" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17585",
"triggerID" : "6451d26be69f5d61b58c08539f0e72b5f99e64cd",
"triggerType" : "PUSH"
}, {
"hash" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17587",
"triggerID" : "54d27f1f45cb117f785a1a2bfcbf0562358e2e13",
"triggerType" : "PUSH"
}, {
"hash" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588",
"triggerID" : "b846e14a7d153415da9392c407019cbc5ce7a5a5",
"triggerType" : "PUSH"
}, {
"hash" : "a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17975",
"triggerID" : "a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b846e14a7d153415da9392c407019cbc5ce7a5a5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17588)
* a8f9f6237ef08cb8b478ea0d0d8bc2a755381e0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17975)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8729: [HUDI-6223] Support Incremental read via Spark SQL table valued function
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8729:
URL: https://github.com/apache/hudi/pull/8729#issuecomment-1549523691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124",
"triggerID" : "9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9713b8d8114a8bc0ba7dc2d5020d75af1c1ad8b1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17124)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org