You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/18 07:28:15 UTC
[GitHub] [hudi] xushiyan opened a new pull request, #5343: fallback to HadoopFsRelation
xushiyan opened a new pull request, #5343:
URL: https://github.com/apache/hudi/pull/5343
If schema on read is not enabled, use `HadoopFsRelation`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5343: fallback to HadoopFsRelation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5343:
URL: https://github.com/apache/hudi/pull/5343#issuecomment-1101202978
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8097",
"triggerID" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"triggerType" : "PUSH"
}, {
"hash" : "300c4f0c2d75619024f06efe764d1cc8990521c3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "300c4f0c2d75619024f06efe764d1cc8990521c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8097)
* 300c4f0c2d75619024f06efe764d1cc8990521c3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5343: fallback to HadoopFsRelation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5343:
URL: https://github.com/apache/hudi/pull/5343#issuecomment-1101299756
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8097",
"triggerID" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"triggerType" : "PUSH"
}, {
"hash" : "300c4f0c2d75619024f06efe764d1cc8990521c3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8099",
"triggerID" : "300c4f0c2d75619024f06efe764d1cc8990521c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 300c4f0c2d75619024f06efe764d1cc8990521c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8099)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5343: fallback to HadoopFsRelation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5343:
URL: https://github.com/apache/hudi/pull/5343#issuecomment-1101220518
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8097",
"triggerID" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"triggerType" : "PUSH"
}, {
"hash" : "300c4f0c2d75619024f06efe764d1cc8990521c3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8099",
"triggerID" : "300c4f0c2d75619024f06efe764d1cc8990521c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8097)
* 300c4f0c2d75619024f06efe764d1cc8990521c3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8099)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #5343: fallback to HadoopFsRelation
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on code in PR #5343:
URL: https://github.com/apache/hudi/pull/5343#discussion_r851966494
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BaseFileOnlyRelation.scala:
##########
@@ -101,4 +102,17 @@ class BaseFileOnlyRelation(sqlContext: SQLContext,
sparkAdapter.getFilePartitions(sparkSession, fileSplits, maxSplitBytes).map(HoodieBaseFileSplit.apply)
}
+
+ def toHadoopFsRelation: HadoopFsRelation = {
+ HadoopFsRelation(
+ fileIndex,
+ fileIndex.partitionSchema,
+ fileIndex.dataSchema,
+ bucketSpec = None,
+ fileFormat = metaClient.getTableConfig.getBaseFileFormat match {
+ case HoodieFileFormat.PARQUET => new ParquetFileFormat
+ case HoodieFileFormat.ORC => new OrcFileFormat
+ },
+ optParams)(sparkSession)
Review Comment:
maybe we need HoodieFileIndex or HoodieROTablePathFilter to filter old parquet file.
how about fallabck to the implement of 0.10.0 https://github.com/apache/hudi/blob/cc3896be2a023f8819883d745178503286ac2ab1/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala#L182
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed pull request #5343: fallback to HadoopFsRelation
Posted by GitBox <gi...@apache.org>.
xushiyan closed pull request #5343: fallback to HadoopFsRelation
URL: https://github.com/apache/hudi/pull/5343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5343: fallback to HadoopFsRelation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5343:
URL: https://github.com/apache/hudi/pull/5343#issuecomment-1101178906
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #5343: fallback to HadoopFsRelation
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #5343:
URL: https://github.com/apache/hudi/pull/5343#issuecomment-1101925730
close in favor of #5352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on a diff in pull request #5343: fallback to HadoopFsRelation
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on code in PR #5343:
URL: https://github.com/apache/hudi/pull/5343#discussion_r852017845
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -98,6 +99,7 @@ class DefaultSource extends RelationProvider
val isBootstrappedTable = metaClient.getTableConfig.getBootstrapBasePath.isPresent
val tableType = metaClient.getTableType
val queryType = parameters(QUERY_TYPE.key)
+ val enableSchemaOnRead = parameters.getOrElse(SCHEMA_EVOLUTION_ENABLE.key, SCHEMA_EVOLUTION_ENABLE.defaultValue.toString).toBoolean
val userSchema = if (schema == null) Option.empty[StructType] else Some(schema)
Review Comment:
maybe it better to use follow code.
```
val enableSchemaOnRead: Boolean = try {
!(new TableSchemaResolver(metaClient).getTableInternalSchemaFromCommitMetadata.orElse(InternalSchema.getEmptyInternalSchema)).isEmptySchema
} catch {
case _: Exception => false
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5343: fallback to HadoopFsRelation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5343:
URL: https://github.com/apache/hudi/pull/5343#issuecomment-1101205346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8097",
"triggerID" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"triggerType" : "PUSH"
}, {
"hash" : "300c4f0c2d75619024f06efe764d1cc8990521c3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "300c4f0c2d75619024f06efe764d1cc8990521c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8097)
* 300c4f0c2d75619024f06efe764d1cc8990521c3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5343: fallback to HadoopFsRelation
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5343:
URL: https://github.com/apache/hudi/pull/5343#issuecomment-1101180671
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8097",
"triggerID" : "0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0354fcbd2b5841c4c403cc1a4f3b5bd0da647bec Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8097)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org