You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/29 07:23:13 UTC
[GitHub] [hudi] JoshuaZhuCN opened a new issue, #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
JoshuaZhuCN opened a new issue, #7322:
URL: https://github.com/apache/hudi/issues/7322
SparkSQL can not read the latest change data without execute "refresh table xxx" after write the data in datasource mode
**To Reproduce**
Steps to reproduce the behavior:
1. run spark-shell and import class
``` import org.apache.spark.sql.SaveMode ```
2. create like this:
```
spark.sql(
s"""|CREATE TABLE IF NOT EXISTS `malawi`.`hudi_0_12_1_spark_test` (
| `id` INT
| ,`name` STRING
| ,`age` INT
| ,`sync_time` TIMESTAMP
|) USING HUDI
|TBLPROPERTIES (
| type = 'mor'
| ,primaryKey = 'id'
| ,preCombineField = 'sync_time'
| ,`hoodie.bucket.index.hash.field` = ''
| ,`hoodie.datasource.write.hive_style_partitioning` = 'false'
| ,`hoodie.table.keygenerator.class`='org.apache.hudi.keygen.ComplexKeyGenerator'
|)
|COMMENT 'hudi_0.12.1_test'""".stripMargin
)
spark.sql(
s"""|create table `malawi`.`hudi_0_12_1_spark_test_rt`
|using hudi
|options(`hoodie.query.as.ro.table` = 'false')
|location 'hdfs://NameNodeService1/hoodie/leqee/malawi/hudi_0_12_1_spark_test';
|""".stripMargin
)
```
3. make test data
```
var dfData = spark.sql(
s"""|select 1 as id,'name1' as name, 18 as age, now() as sync_time
| union all
|select 2 as id,'name2' as name, 22 as age, now() as sync_time
| union all
|select 3 as id,'name3' as name, 23 as age, now() as sync_time
|""".stripMargin
)
var dfData2 = spark.sql(
s"""|select 4 as id,'name1' as name, 18 as age, now() as sync_time
|""".stripMargin
)
```
4. make hudi datasource options
```
var hoodieProp = Map("hoodie.table.name" -> "hudi_0_12_1_spark_test", "hoodie.datasource.write.operation"->"upsert", "hoodie.datasource.write.recordkey.field" -> "id", "hoodie.datasource.write.keygenerator.class"->"org.apache.hudi.keygen.ComplexKeyGenerator", "hoodie.datasource.write.partitionpath.field"->"", "hoodie.datasource.write.precombine.field"->"sync_time", "hoodie.metadata.enable"->"true", "hoodie.upsert.shuffle.parallelism"->"10", "hoodie.embed.timeline.server"->"false")
```
5. write data fist time
```
dfData.write.format("org.apache.hudi").options(hoodieProp).mode(SaveMode.Append).save("hdfs://xxx/malawi/hudi_0_12_1_spark_test")
```
6. query in spark sql
```
spark.sql(s"""|select *
| from (
| select 'ori' as flag,a.* from `malawi`.`hudi_0_12_1_spark_test` a
| union all
| select '_rt' as flag,b.* from `malawi`.`hudi_0_12_1_spark_test_rt` b
| ) t
|order by t.id asc, t.flag asc""".stripMargin
).show(false)
```
7. write data second time
```
dfData2.write.format("org.apache.hudi").options(hoodieProp).mode(SaveMode.Append).save("hdfs://xxx/malawi/hudi_0_12_1_spark_test")
```
8. repeat step 6
the data(id = 4) should be queried out in _rt table but it is not
9. refresh table
```
spark.sql("REFRESH TABLE `malawi`.`hudi_0_12_1_spark_test`")
```
10. repeat step 6
**Environment Description**
* Hudi version : 0.12.1
* Spark version : 3.1.3
* Hive version : 3.1.1
* Hadoop version : 3.1.0
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1350309180
@alexeykudinkin Let's resolve the related issues all in this one. Re-open it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1349678545
@JoshuaZhuCN the reason why it's not refreshing is b/c you're writing into a table by path and not by table identifier (ie `db.table`) -- therefore it bypasses the Catalog and goes straight to Hudi's Data Source which is not going to be refreshing the Spark caches.
This is not an issue and is an expected behavior. To address try writing into the table by its id (which in turn will involve the catalog refresh)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1350235937
> @alexeykudinkin i don't understand what "write into the table by its id" means, just using sql like insert into/update/delete from db.table to write data?
Correct. You can do the same from Spark DS.
> @alexeykudinkin At present, the problem I encounter is not only that the Spark datasource cannot be read after it is written, but also that the Spark sql cannot be read after it is written by Flink using hive sync. In other words, the SparkSQL query can not immediately read new data in any other way except by writing data in SQL. Therefore, I think this is a problem that needs to be solved
Interesting. Can you please create another issue specifically for this one as this hardly could be related?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin closed issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
alexeykudinkin closed issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
URL: https://github.com/apache/hudi/issues/7322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1373064499
@JoshuaZhuCN you can try this issues around by actually switching off Spark's file relation cache by setting `spark.sql.filesourceTableRelationCacheSize=0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1372475071
@JoshuaZhuCN thanks for creating #7452.
Closing this now, as this is not really a Hudi issue, but rather the way Spark is used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JoshuaZhuCN commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
JoshuaZhuCN commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1350194639
> @JoshuaZhuCN the reason why it's not refreshing is b/c you're writing into a table by path and not by table identifier (ie `db.table`) -- therefore it bypasses the Catalog and goes straight to Hudi's Data Source which is not going to be refreshing the Spark caches.
>
> This is not an issue and is an expected behavior. To address try writing into the table by its id (which in turn will involve the catalog refresh)
@alexeykudinkin i don`t understand what "write into the table by its id" means, just using sql like `insert into/update/delete from db.table` to write date?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JoshuaZhuCN commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
JoshuaZhuCN commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1350206823
@alexeykudinkin At present, the problem I encounter is not only that the Spark datasource cannot be read after it is written, but also that the Spark sql cannot be read after it is written by Flink using hive sync. In other words, the SparkSQL query can not immediately read new data in any other way except by writing data in SQL. Therefore, I think this is a problem that needs to be solved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JoshuaZhuCN commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
JoshuaZhuCN commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1350344575
> > @alexeykudinkin At present, the problem I encounter is not only that the Spark datasource cannot be read after it is written, but also that the Spark sql cannot be read after it is written by Flink using hive sync. In other words, the SparkSQL query can not immediately read new data in any other way except by writing data in SQL. Therefore, I think this is a problem that needs to be solved
>
> Interesting. Can you please create another issue specifically for this one as this hardly could be related?
@alexeykudinkin here https://github.com/apache/hudi/issues/7452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
boneanxs commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1330618582
Yes, I think @danny0405 is right. Like Spark file listing, hudi also cache file status to avoid call filesystem repeatedly when listing files. And the cache will be update when you call `refresh` explicitly.
But I'm thinking whether we can hold the number of complete commits as a variable, and decide whether to use the cache or not by comparing if that num is changed or not.[This cannot work if the relation changed to HadoopRelation, `ToHadoopRelation` use Spark InmemoryFileIndex]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1350336358
> @alexeykudinkin I think the query engine should not limit the writing way for querying data. Even for the tables created by Spakrsql, the query engine should be able to query new data regardless of the way in which the data is written by spark datasource, spark sql, java client, flink sql, and flink stream api, without requiring users to do additional operations for different writing methods when using the query engine.
This is not a limitation of the query engine this is a limitation of how you're using the query engine -- when writing to a table specified as a path following issues are at play
1. Spark SQL will cache the Relation w/in the session cache when queried
2. When writing to a table identified by a full path, rather than a name and Spark has no way to invalidate the SQL session cache (since it doesn't have the table identifier)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JoshuaZhuCN commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
JoshuaZhuCN commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1340612815
@alexeykudinkin Hi，is there any way to solve this problem, such as closing some parameter settings? In addition to manually refreshing the table (because this means that a large number of business codes need to be modified), there is no temporary closure measure. The bug that must be solved by modifying the business code is a destructive defect, which directly affects the upgrade and use. This problem should be repaired through the patch version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JoshuaZhuCN commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
JoshuaZhuCN commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1350311469
> > @alexeykudinkin i don't understand what "write into the table by its id" means, just using sql like insert into/update/delete from db.table to write data?
>
> Correct. You can do the same from Spark DS.
>
@alexeykudinkin I think the query engine should not limit the writing method for querying data. Even for the tables created by Spakrsql, the query engine should be able to query new data regardless of the way in which the data is written in the spark datasource, spark sql, java client, flash sql, and flash stream apis, without requiring users to do additional operations for different writing methods when using the query engine
> > @alexeykudinkin At present, the problem I encounter is not only that the Spark datasource cannot be read after it is written, but also that the Spark sql cannot be read after it is written by Flink using hive sync. In other words, the SparkSQL query can not immediately read new data in any other way except by writing data in SQL. Therefore, I think this is a problem that needs to be solved
>
> Interesting. Can you please create another issue specifically for this one as this hardly could be related?
I'll verify it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin closed issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
alexeykudinkin closed issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
URL: https://github.com/apache/hudi/issues/7322
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1350331029
> @alexeykudinkin Let's resolve the related issues all in this one. Re-open it.
Can you elaborate why you think these issues are related?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #7322: [SUPPORT][HELP] SparkSQL can not read the latest change data without execute "refresh table xxx"
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #7322:
URL: https://github.com/apache/hudi/issues/7322#issuecomment-1330278407
@alexeykudinkin Can you take a look, is this related with the FileIndex cache ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org