You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "amrishlal (via GitHub)" <gi...@apache.org> on 2023/07/11 18:24:42 UTC
[GitHub] [hudi] amrishlal opened a new pull request, #9169: [HUDI-6515] Commenting out failing test case.
amrishlal opened a new pull request, #9169:
URL: https://github.com/apache/hudi/pull/9169
### Change Logs
The test case `Test Bulk Insert Into Bucket Index Table` in `TestInsertTable.scala` is failing as a result of changes made in ##9156. Commenting out the test case to unblock to the build.
### Impact
None
### Risk level (write none, low medium or high below)
None
### Documentation Update
None
### Contributor's checklist
- [X] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [X] Change Logs and Impact were stated clearly
- [X] Adequate tests were added if applicable
- [X] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on a diff in pull request #9169: [HUDI-6515] Commenting out failing test case.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on code in PR #9169:
URL: https://github.com/apache/hudi/pull/9169#discussion_r1260152774
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -1191,88 +1191,68 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
}
}
- test("Test Bulk Insert Into Bucket Index Table") {
- withSQLConf("hoodie.datasource.write.operation" -> "bulk_insert", "hoodie.bulkinsert.shuffle.parallelism" -> "1") {
- Seq("mor", "cow").foreach { tableType =>
- Seq("true", "false").foreach { bulkInsertAsRow =>
- withTempDir { tmp =>
- val tableName = generateTableName
- // Create a partitioned table
- spark.sql(
- s"""
- |create table $tableName (
- | id int,
- | dt string,
- | name string,
- | price double,
- | ts long
- |) using hudi
- | tblproperties (
- | primaryKey = 'id,name',
- | type = '$tableType',
- | preCombineField = 'ts',
- | hoodie.index.type = 'BUCKET',
- | hoodie.bucket.index.hash.field = 'id,name',
- | hoodie.datasource.write.row.writer.enable = '$bulkInsertAsRow')
- | partitioned by (dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
-
- // Note: Do not write the field alias, the partition field must be placed last.
- spark.sql(
- s"""
- | insert into $tableName values
- | (1, 'a1,1', 10, 1000, "2021-01-05"),
- | (2, 'a2', 20, 2000, "2021-01-06"),
- | (3, 'a3,3', 30, 3000, "2021-01-07")
- """.stripMargin)
-
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1,1", 10.0, 1000, "2021-01-05"),
- Seq(2, "a2", 20.0, 2000, "2021-01-06"),
- Seq(3, "a3,3", 30.0, 3000, "2021-01-07")
- )
-
- spark.sql(
- s"""
- | insert into $tableName values
- | (1, 'a1', 10, 1000, "2021-01-05"),
- | (3, "a3", 30, 3000, "2021-01-07")
- """.stripMargin)
-
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1,1", 10.0, 1000, "2021-01-05"),
- Seq(1, "a1", 10.0, 1000, "2021-01-05"),
- Seq(2, "a2", 20.0, 2000, "2021-01-06"),
- Seq(3, "a3,3", 30.0, 3000, "2021-01-07"),
- Seq(3, "a3", 30.0, 3000, "2021-01-07")
- )
-
- // there are two files in partition(dt = '2021-01-05')
- checkAnswer(s"select count(distinct _hoodie_file_name) from $tableName where dt = '2021-01-05'")(
- Seq(2)
- )
-
- // would generate 6 other files in partition(dt = '2021-01-05')
- spark.sql(
- s"""
- | insert into $tableName values
- | (4, 'a1,1', 10, 1000, "2021-01-05"),
- | (5, 'a1,1', 10, 1000, "2021-01-05"),
- | (6, 'a1,1', 10, 1000, "2021-01-05"),
- | (7, 'a1,1', 10, 1000, "2021-01-05"),
- | (8, 'a1,1', 10, 1000, "2021-01-05"),
- | (9, 'a3,3', 30, 3000, "2021-01-05")
- """.stripMargin)
-
- checkAnswer(s"select count(distinct _hoodie_file_name) from $tableName where dt = '2021-01-05'")(
- Seq(8)
- )
- }
- }
- }
- }
- }
+ // FAILING TEST CASE. DO NOT UNCOMMENT BEFORE FIXING THE TEST CASE (see https://github.com/apache/hudi/pull/9156)
Review Comment:
Fixed using following syntax/structure
```
/** Ignore failing test case (see HUDI-6521 for more details) */
ignore("Test Bulk Insert Into Bucket Index Table") {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9169: [HUDI-6515] Commenting out failing test case.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9169:
URL: https://github.com/apache/hudi/pull/9169#discussion_r1260124577
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -1191,88 +1191,68 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
}
}
- test("Test Bulk Insert Into Bucket Index Table") {
- withSQLConf("hoodie.datasource.write.operation" -> "bulk_insert", "hoodie.bulkinsert.shuffle.parallelism" -> "1") {
- Seq("mor", "cow").foreach { tableType =>
- Seq("true", "false").foreach { bulkInsertAsRow =>
- withTempDir { tmp =>
- val tableName = generateTableName
- // Create a partitioned table
- spark.sql(
- s"""
- |create table $tableName (
- | id int,
- | dt string,
- | name string,
- | price double,
- | ts long
- |) using hudi
- | tblproperties (
- | primaryKey = 'id,name',
- | type = '$tableType',
- | preCombineField = 'ts',
- | hoodie.index.type = 'BUCKET',
- | hoodie.bucket.index.hash.field = 'id,name',
- | hoodie.datasource.write.row.writer.enable = '$bulkInsertAsRow')
- | partitioned by (dt)
- | location '${tmp.getCanonicalPath}'
- """.stripMargin)
-
- // Note: Do not write the field alias, the partition field must be placed last.
- spark.sql(
- s"""
- | insert into $tableName values
- | (1, 'a1,1', 10, 1000, "2021-01-05"),
- | (2, 'a2', 20, 2000, "2021-01-06"),
- | (3, 'a3,3', 30, 3000, "2021-01-07")
- """.stripMargin)
-
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1,1", 10.0, 1000, "2021-01-05"),
- Seq(2, "a2", 20.0, 2000, "2021-01-06"),
- Seq(3, "a3,3", 30.0, 3000, "2021-01-07")
- )
-
- spark.sql(
- s"""
- | insert into $tableName values
- | (1, 'a1', 10, 1000, "2021-01-05"),
- | (3, "a3", 30, 3000, "2021-01-07")
- """.stripMargin)
-
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1,1", 10.0, 1000, "2021-01-05"),
- Seq(1, "a1", 10.0, 1000, "2021-01-05"),
- Seq(2, "a2", 20.0, 2000, "2021-01-06"),
- Seq(3, "a3,3", 30.0, 3000, "2021-01-07"),
- Seq(3, "a3", 30.0, 3000, "2021-01-07")
- )
-
- // there are two files in partition(dt = '2021-01-05')
- checkAnswer(s"select count(distinct _hoodie_file_name) from $tableName where dt = '2021-01-05'")(
- Seq(2)
- )
-
- // would generate 6 other files in partition(dt = '2021-01-05')
- spark.sql(
- s"""
- | insert into $tableName values
- | (4, 'a1,1', 10, 1000, "2021-01-05"),
- | (5, 'a1,1', 10, 1000, "2021-01-05"),
- | (6, 'a1,1', 10, 1000, "2021-01-05"),
- | (7, 'a1,1', 10, 1000, "2021-01-05"),
- | (8, 'a1,1', 10, 1000, "2021-01-05"),
- | (9, 'a3,3', 30, 3000, "2021-01-05")
- """.stripMargin)
-
- checkAnswer(s"select count(distinct _hoodie_file_name) from $tableName where dt = '2021-01-05'")(
- Seq(8)
- )
- }
- }
- }
- }
- }
+ // FAILING TEST CASE. DO NOT UNCOMMENT BEFORE FIXING THE TEST CASE (see https://github.com/apache/hudi/pull/9156)
Review Comment:
This is now how we disable tests. we use "Disabled" tag to disable tests. we need to add a hudi jira adding links to failures and stacktraces may be and then add jira no in the Disabled annotation. you can find examples in our test suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9169: [HUDI-6521] Disable failing test case.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9169:
URL: https://github.com/apache/hudi/pull/9169#issuecomment-1631627132
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccae5025f90656628abc0b75d452d4b3ad1dab6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18496",
"triggerID" : "7ccae5025f90656628abc0b75d452d4b3ad1dab6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ccae5025f90656628abc0b75d452d4b3ad1dab6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18496)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] amrishlal commented on pull request #9169: [HUDI-6521] Disable failing test case.
Posted by "amrishlal (via GitHub)" <gi...@apache.org>.
amrishlal commented on PR #9169:
URL: https://github.com/apache/hudi/pull/9169#issuecomment-1631635510
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #9169: [HUDI-6521] Disable failing test case.
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #9169:
URL: https://github.com/apache/hudi/pull/9169#issuecomment-1631745271
Hi @amrishlal and @nsivabalan, sorry for the test case failure introduced. I have raised #9163 and fixed ci. Should we just revert this pr and enable the test again? Thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9169: [HUDI-6521] Disable failing test case.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9169:
URL: https://github.com/apache/hudi/pull/9169#issuecomment-1631382485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccae5025f90656628abc0b75d452d4b3ad1dab6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18496",
"triggerID" : "7ccae5025f90656628abc0b75d452d4b3ad1dab6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ccae5025f90656628abc0b75d452d4b3ad1dab6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18496)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9169: [HUDI-6521] Disable failing test case.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9169:
URL: https://github.com/apache/hudi/pull/9169#issuecomment-1631371916
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7ccae5025f90656628abc0b75d452d4b3ad1dab6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7ccae5025f90656628abc0b75d452d4b3ad1dab6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7ccae5025f90656628abc0b75d452d4b3ad1dab6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #9169: [HUDI-6521] Disable failing test case.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #9169:
URL: https://github.com/apache/hudi/pull/9169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org