You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/29 18:03:13 UTC
[GitHub] [hudi] BenjMaq opened a new issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
BenjMaq opened a new issue #4154:
URL: https://github.com/apache/hudi/issues/4154
**Describe the problem you faced**
_Disclaimer: Creating and inserting into external hive tables stored on S3._
- The `INSERT OVERWRITE` operation does not work when using spark SQL. When running `INSERT OVERWRITE` on an existing partition, the parquet files get correctly created (I can see them in S3) but the partition (metadata?) does not get updated. When selecting from the same table, the old files are still being queried and not the new ones.
- Also, when running `INSERT OVERWRITE` on an empty table, the parquet files get correctly created as well (I see them in S3) but when selecting from the table, an empty result set is returned.
**To Reproduce**
1. Create an external table
```
CREATE TABLE IF NOT EXISTS <schema>.<table_name> (
id bigint,
name string,
dt string
) USING hudi
LOCATION 's3a://<bucket>/<object>'
OPTIONS (
type = 'cow'
)
PARTITIONED by (dt);
```
2. Insert some data
```
insert into <schema>.<table_name>
values
(1, 'a1', '2021-11-29'),
(2, 'a2', '2021-11-29')
;
```
3. Check the table (I use Presto/Hive)
<body>
_hoodie_commit_time | _hoodie_commit_seqno | _hoodie_record_key | _hoodie_partition_path | _hoodie_file_name | id | name | dt
-- | -- | -- | -- | -- | -- | -- | --
20211129094605 | 20211129094605_0_1 | 65ff84b9-9733-48f6-bb79-d22b239f81c1 | dt=2021-11-29 | e80b1f6e-d066-416f-8442-95c4172c94cf-0_0-5-8_20211129094605.parquet | 1 | a1 | 2021-11-29
20211129094605 | 20211129094605_0_2 | 73712448-b8a5-4ec1-a363-f667a520261b | dt=2021-11-29 | e80b1f6e-d066-416f-8442-95c4172c94cf-0_0-5-8_20211129094605.parquet | 2 | a2 | 2021-11-29
</body>
4. Run `insert overwrite` statement
```
insert overwrite table public.test_overwrite_bm
values
(3, 'a3', '2021-11-29'),
(4, 'a4', '2021-11-29')
;
```
4. Check the table again => it's the same as before. The `insert overwrite` statement did not have any effect
<body>
_hoodie_commit_time | _hoodie_commit_seqno | _hoodie_record_key | _hoodie_partition_path | _hoodie_file_name | id | name | dt
-- | -- | -- | -- | -- | -- | -- | --
20211129094605 | 20211129094605_0_1 | 65ff84b9-9733-48f6-bb79-d22b239f81c1 | dt=2021-11-29 | e80b1f6e-d066-416f-8442-95c4172c94cf-0_0-5-8_20211129094605.parquet | 1 | a1 | 2021-11-29
20211129094605 | 20211129094605_0_2 | 73712448-b8a5-4ec1-a363-f667a520261b | dt=2021-11-29 | e80b1f6e-d066-416f-8442-95c4172c94cf-0_0-5-8_20211129094605.parquet | 2 | a2 | 2021-11-29
</body>
**Expected behavior**
I would expect values of `id` and `name` in the above example to become `3, 4` and `a3, a3`, respectively.
**Environment Description**
* Hudi version : 0.9.0
* Spark version : 2.4.4
* Hive version : 2.3.5
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : No
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
YannByron edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1002370436
@nsivabalan @BenjMaq
I use the basically same commands, in Hudi0.9 + Spark2.4.4.
```
CREATE TABLE IF NOT EXISTS test_overwrite (
id bigint,
name string,
dt string
) USING hudi
LOCATION 'file:///tmp/hudi/test_overwrite'
OPTIONS (
type = 'cow'
)
PARTITIONED by (dt);
insert into test_overwrite
values
(1, 'a1', '2021-11-29'),
(2, 'a2', '2021-11-29')
;
insert overwrite table test_overwrite
values
(3, 'a3', '2021-11-29'),
(4, 'a4', '2021-11-29')
;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4154:
URL: https://github.com/apache/hudi/issues/4154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4154:
URL: https://github.com/apache/hudi/issues/4154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1022077497
Hi team @nsivabalan, I only tested with Hive yesterday and it worked. However, testing with Presto `0.247` now and I still cannot read the table correctly.
I did some research and found out Hudi was bumped to `0.9.0` in PrestoDB `0.265` ([PR](https://github.com/prestodb/presto/commit/b52db2c4c2baa720489ae3908acd9303a41081fc), [release notes](https://prestodb.io/docs/current/release/release-0.265.html)). For Presto `<0.265`, Hudi `0.5.3` is installed.
I believe that's where the problem is. My version of Presto only has Hudi `0.5.3` and thus I cannot correctly read replacecommit files.
The [Hudi docs](https://hudi.apache.org/docs/query_engine_setup#PrestoDB) do not mention anything about this issue. It only says "Presto >= 0.240 | No action needed. Hudi 0.5.3 version is a compile time dependency."
Am I correct assuming Hudi `0.5.3` cannot correctly read replacecommit files? If yes, could we maybe update the Docs to reflect this?
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
codope commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1004760934
@BenjMaq That's right. If you did that hive/presto setup once it's enough. We don't need to sync every commit. As for the difference between commit and replacecommit, that is expected for insert overwrite.
Do you mind trying the latest master? I can't repro the issue in latest master. Meanwhile, i'll try with hudi 0.9.0.
> Also, I'm not sure what hudi-hive-sync is and what it is used for? I've never needed to run it for insert/delete/update operations. But maybe I'm missing something?
`hudi-hive-sync` module provides you `run_sync_tool` which sync the hudi table to hive metastore and exposes it to be queried from other engines like presto/trino. You can read more about it [here](https://hudi.apache.org/docs/syncing_metastore) and see that in action in a quick [docker demo](https://hudi.apache.org/docs/docker_demo).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1021303523
Hi again,
Still trying to understand the issue :)
Here's the create table statement I run to create the table using `spark-sql`:
```
CREATE TABLE IF NOT EXISTS public.test_partitioned (
id bigint,
name string,
start_date string,
dt string
) USING hudi
LOCATION 's3a://<s3_bucket>/hudi_data_lake/test_partitioned'
OPTIONS (
type = 'cow',
primaryKey = 'id',
preCombineField = 'start_date'
)
PARTITIONED BY (dt);
```
and here's the DDL when I run `show create table public.test_partitioned` in Hive cli:
```
CREATE EXTERNAL TABLE `public.test_partitioned`(
`_hoodie_commit_time` string,
`_hoodie_commit_seqno` string,
`_hoodie_record_key` string,
`_hoodie_partition_path` string,
`_hoodie_file_name` string,
`id` bigint,
`name` string,
`start_date` string)
PARTITIONED BY (
`dt` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
'path'='s3a://<s3_bucket>/hudi_data_lake/test_partitioned',
'preCombineField'='start_date',
'primaryKey'='id',
'type'='cow')
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.HoodieParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
's3a://<s3_bucket>/hudi_data_lake/test_partitioned'
TBLPROPERTIES (
'last_commit_time_sync'='20220125071309',
'spark.sql.create.version'='2.4.4-20210211',
'spark.sql.sources.provider'='hudi',
'spark.sql.sources.schema.numPartCols'='1',
'spark.sql.sources.schema.numParts'='1',
'spark.sql.sources.schema.part.0'='{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"start_date","type":"string","nullable":true,"metadata":{}},{"name":"dt","type":"string","nullable":true,"metadata":{}}]}',
'spark.sql.sources.schema.partCol.0'='dt',
'transient_lastDdlTime'='1643122431')
```
Are you maybe seeing something that looks off? I see for example
```
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.HoodieParquetInputFormat'
```
which is neither `HiveInputFormat` nor `HoodieCombineHiveInputFormat` mentioned by @nsivabalan. Is that normal?
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-994623507
@YannByron I use the same Spark and Hudi as you and I still have this issue in `0.10.0`.
I see the parquet files being correctly created but the table still points to the old files (not the new ones created...).
The `20211215020719844.replacecommit` file is also correctly created, pointing to the new parquet files.
```
{
"partitionToWriteStats" : {
"dt=2021-12-14" : [ {
"fileId" : "655d468d-bb0d-44e2-9115-66d33e84a9a2-0",
"path" : "dt=2021-12-14/655d468d-bb0d-44e2-9115-66d33e84a9a2-0_0-12-1006_20211215020719844.parquet",
"prevCommit" : "null",
"numWrites" : 2,
"numDeletes" : 0,
"numUpdateWrites" : 0,
"numInserts" : 2,
"totalWriteBytes" : 435257,
"totalWriteErrors" : 0,
"tempPath" : null,
"partitionPath" : "dt=2021-12-14",
"totalLogRecords" : 0,
"totalLogFilesCompacted" : 0,
"totalLogSizeCompacted" : 0,
"totalUpdatedRecordsCompacted" : 0,
"totalLogBlocks" : 0,
"totalCorruptLogBlock" : 0,
"totalRollbackBlocks" : 0,
"fileSizeInBytes" : 435257,
"minEventTime" : null,
"maxEventTime" : null
} ]
},
"compacted" : false,
"extraMetadata" : {
"schema" : "{\"type\":\"record\",\"name\":\"test_partitioned_record\",\"namespace\":\"hoodie.test_partitioned\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"long\"],\"default\":null},{\"name\":\"name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"start_date\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"dt\",\"type\":[\"null\",\"string\"],\"default\":null}]}"
},
"operationType" : "INSERT_OVERWRITE",
"partitionToReplaceFileIds" : {
"dt=2021-12-14" : [ "51a58eaa-f7b1-4234-876a-65ddd6a28f08-0" ]
},
"totalRecordsDeleted" : 0,
"totalLogRecordsCompacted" : 0,
"totalLogFilesCompacted" : 0,
"totalCompactedRecordsUpdated" : 0,
"totalLogFilesSize" : 0,
"totalScanTime" : 0,
"totalCreateTime" : 2219,
"totalUpsertTime" : 0,
"minAndMaxEventTime" : {
"Optional.empty" : {
"val" : null,
"present" : false
}
},
"fileIdAndRelativePaths" : {
"655d468d-bb0d-44e2-9115-66d33e84a9a2-0" : "dt=2021-12-14/655d468d-bb0d-44e2-9115-66d33e84a9a2-0_0-12-1006_20211215020719844.parquet"
},
"writePartitionPaths" : [ "dt=2021-12-14" ]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-997125124
@YannByron @danny0405 : Can either of you triage this. We might need a fix if its a bug. Feel free to file a tracking jira and work towards it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1004675484
@codope Hive is automatically synced. It works correctly for inserts, updates, deletes. I followed [these steps](https://hudi.apache.org/docs/query_engine_setup#hive) initially for Presto and Hive and it has always worked without the need for an extra manual sync. I'm just wondering why `insert overwrite` does not work the same way?
I suspect there is something wrong with the `*replacecommit` files. When I insert/update, the files are named `*commit` while when I overwrite they are named `*replacecommit`. It seems only the `*replacecommit` are not being correctly read via Presto/Hive.
Also, I'm not sure what `hudi-hive-sync` is and what it is used for? I've never needed to run it for insert/delete/update operations. But maybe I'm missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
YannByron edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1002370436
@nsivabalan @BenjMaq
I use the basically same commands, in Hudi0.9 + Spark2.4.4.
```CREATE TABLE IF NOT EXISTS test_overwrite (
id bigint,
name string,
dt string
) USING hudi
LOCATION 'file:///tmp/hudi/test_overwrite'
OPTIONS (
type = 'cow'
)
PARTITIONED by (dt);
insert into test_overwrite
values
(1, 'a1', '2021-11-29'),
(2, 'a2', '2021-11-29')
;
insert overwrite table test_overwrite
values
(3, 'a3', '2021-11-29'),
(4, 'a4', '2021-11-29')
;```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1021348122
thanks for updating!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1014468802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1021327116
I found the problem. The JAR in my Hive aux path was outdated. I replaced it with the latest version and all was good. Sorry for the inconvenience, and thank you so much for supporting and trying to find the error!
Good to close on my end!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-994623507
@YannByron I use the same Spark and Hudi as you and I still have this issue in `0.10.0`.
I see the parquet files being correctly created but the table still points to the old files (not the new ones created...).
The `20211215020719844.replacecommit` file is also correctly created, pointing to the new parquet files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-994697240
Our flink sql on 0.10.0 is also failing, i have checked the metadata is correct, i suspect there was something wrong with the filesystem view strategy for replace commit. Somehow, it is filtered out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1037241618
Hi team, wondering if anyone was able to corroborate (or refute) my above statement.
Also wondering if this issue could be re-opened?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1021348122
thanks for updating!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1022077497
Hi team, I only tested with Hive yesterday and it worked. However, testing with Presto `0.247` now and I still cannot read the table correctly.
I did some research and found out Hudi was bumped to `0.9.0` in PrestoDB `0.265` ([PR](https://github.com/prestodb/presto/commit/b52db2c4c2baa720489ae3908acd9303a41081fc), [release notes](https://prestodb.io/docs/current/release/release-0.265.html)). For Presto `<0.265`, Hudi `0.5.3` is installed.
I believe that's where the problem is. My version of Presto only has Hudi `0.5.3` and thus I cannot correctly read replacecommit files.
The Hudi docs](https://hudi.apache.org/docs/query_engine_setup#PrestoDB) do not mention anything about this issue. It only says "Presto >= 0.240 | No action needed. Hudi 0.5.3 version is a compile time dependency."
Am I correct assuming Hudi `0.5.3` cannot correctly read replacecommit files? If yes, could we maybe update the Docs to reflect this?
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1040421711
yes, we hudi 0.5.3 cannot read replace commit files. it was added only in 0.7.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1004675484
@codope Hive is automatically synced. It works correctly for inserts, updates, deletes. I followed [these steps](https://hudi.apache.org/docs/query_engine_setup#hive) initially for Presto and Hive and it has always worked without the need for an extra manual sync. I'm just wondering why `insert overwrite` does not work the same way?
I suspect there is something wrong with the `*replacecommit` files. When I insert/update, the files are named `*commit` while when I overwrite they are named `*replacecommit`. It seems only the `*replacecommit` are not being correctly read via Presto/Hive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1025422834
FYI @codope, can you take this please?
> Hi team @nsivabalan, I only tested with Hive yesterday and it worked. However, testing with Presto `0.247` now and I still cannot read the table correctly.
>
> I did some research and found out Hudi was bumped to `0.9.0` in PrestoDB `0.265` ([PR](https://github.com/prestodb/presto/commit/b52db2c4c2baa720489ae3908acd9303a41081fc), [release notes](https://prestodb.io/docs/current/release/release-0.265.html)). For Presto `<0.265`, Hudi `0.5.3` is installed.
>
> I believe that's where the problem is. My version of Presto only has Hudi `0.5.3` and thus I cannot correctly read replacecommit files.
>
> The [Hudi docs](https://hudi.apache.org/docs/query_engine_setup#PrestoDB) do not mention anything about this issue. It only says "Presto >= 0.240 | No action needed. Hudi 0.5.3 version is a compile time dependency."
>
> Am I correct assuming Hudi `0.5.3` cannot correctly read replacecommit files? If yes, could we maybe update the Docs to reflect this?
>
> Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1002288792
@YannByron : can you post your playbook here please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
codope commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1004559852
@BenjMaq How did you sync to hive? My hunch is something went wrong while hive syncing. Using latest master and the script that @YannByron shared, I ran hive sync and could see the latest data from both hive and presto.
Gist: https://gist.github.com/codope/d42a6fb85ef081b3bcd94d4897a4481d
Hoodie Timeline:
```
root@adhoc-1:/opt# hadoop fs -ls /tmp/hudi/test_overwrite1/.hoodie
Found 11 items
drwxr-xr-x - root supergroup 0 2022-01-04 06:20 /tmp/hudi/test_overwrite1/.hoodie/.aux
drwxr-xr-x - root supergroup 0 2022-01-04 06:21 /tmp/hudi/test_overwrite1/.hoodie/.temp
-rw-r--r-- 1 root supergroup 1791 2022-01-04 06:21 /tmp/hudi/test_overwrite1/.hoodie/20220104062057294.commit
-rw-r--r-- 1 root supergroup 0 2022-01-04 06:20 /tmp/hudi/test_overwrite1/.hoodie/20220104062057294.commit.requested
-rw-r--r-- 1 root supergroup 1194 2022-01-04 06:21 /tmp/hudi/test_overwrite1/.hoodie/20220104062057294.inflight
-rw-r--r-- 1 root supergroup 1907 2022-01-04 06:21 /tmp/hudi/test_overwrite1/.hoodie/20220104062117644.replacecommit
-rw-r--r-- 1 root supergroup 1204 2022-01-04 06:21 /tmp/hudi/test_overwrite1/.hoodie/20220104062117644.replacecommit.inflight
-rw-r--r-- 1 root supergroup 0 2022-01-04 06:21 /tmp/hudi/test_overwrite1/.hoodie/20220104062117644.replacecommit.requested
drwxr-xr-x - root supergroup 0 2022-01-04 06:20 /tmp/hudi/test_overwrite1/.hoodie/archived
-rw-r--r-- 1 root supergroup 985 2022-01-04 06:20 /tmp/hudi/test_overwrite1/.hoodie/hoodie.properties
drwxr-xr-x - root supergroup 0 2022-01-04 06:21 /tmp/hudi/test_overwrite1/.hoodie/metadata
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1040421711
yes, we hudi 0.5.3 cannot read replace commit files. it was added only in 0.7.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1004148341
Hi everyone, sorry for the late reply as I was on holidays.
I tried again after @YannByron 's comment but this time I tried reading the files using Scala (`spark.read.format("hudi")`) and not Presto / Hive... and I was able to see the **correct** data! I then tried again using Presto / Hive, and I still saw the **old** data.
**So I believe it has something to do with Presto / Hive not being able to read the latest metadata of the table.** Would that be possible? Would something be missing in the Presto / Hive read implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
YannByron commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1002370436
@nsivabalan @BenjMaq
I use the basically same commands, in Hudi0.9 + Spark2.4.4.
`CREATE TABLE IF NOT EXISTS test_overwrite (
id bigint,
name string,
dt string
) USING hudi
LOCATION 'file:///tmp/hudi/test_overwrite'
OPTIONS (
type = 'cow'
)
PARTITIONED by (dt);
insert into test_overwrite
values
(1, 'a1', '2021-11-29'),
(2, 'a2', '2021-11-29')
;
insert overwrite table test_overwrite
values
(3, 'a3', '2021-11-29'),
(4, 'a4', '2021-11-29')
;`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1021303523
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1018636053
@nsivabalan I'm not specifying any input format when I query using Presto or Hive. I followed [these steps](https://hudi.apache.org/docs/query_engine_setup#hive) initially for Presto and Hive and it has always worked without the need to specify any input format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1021303523
Hi again,
Still trying to understand the issue :)
Here's the create table statement I run to create the table using `spark-sql`:
```
CREATE TABLE IF NOT EXISTS public.test_partitioned (
id bigint,
name string,
start_date string,
dt string
) USING hudi
LOCATION 's3a://<s3_bucket>/hudi_data_lake/test_partitioned'
OPTIONS (
type = 'cow',
primaryKey = 'id',
preCombineField = 'start_date'
)
PARTITIONED BY (dt);
```
and here's the DDL when I run `show create table public.test_partitioned` in Hive cli:
```
CREATE EXTERNAL TABLE `public.test_partitioned`(
`_hoodie_commit_time` string,
`_hoodie_commit_seqno` string,
`_hoodie_record_key` string,
`_hoodie_partition_path` string,
`_hoodie_file_name` string,
`id` bigint,
`name` string,
`start_date` string)
PARTITIONED BY (
`dt` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
'path'='s3a://<s3_bucket>/hudi_data_lake/test_partitioned',
'preCombineField'='start_date',
'primaryKey'='id',
'type'='cow')
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.HoodieParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
's3a://<s3_bucket>/hudi_data_lake/test_partitioned'
TBLPROPERTIES (
'last_commit_time_sync'='20220125071309',
'spark.sql.create.version'='2.4.4-20210211',
'spark.sql.sources.provider'='hudi',
'spark.sql.sources.schema.numPartCols'='1',
'spark.sql.sources.schema.numParts'='1',
'spark.sql.sources.schema.part.0'='{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"start_date","type":"string","nullable":true,"metadata":{}},{"name":"dt","type":"string","nullable":true,"metadata":{}}]}',
'spark.sql.sources.schema.partCol.0'='dt',
'transient_lastDdlTime'='1643122431')
```
Are you maybe seeing something that looks off? I see for example
```
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.HoodieParquetInputFormat'
```
which is neither `HiveInputFormat` nor `HoodieCombineHiveInputFormat` mentioned by @nsivabalan. Is that normal? As mentioned above, I've followed the steps in https://hudi.apache.org/docs/query_engine_setup#PrestoDB to be able to read from Presto and Hive.
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1022077497
Hi team @nsivabalan, I only tested with Hive yesterday and it worked. However, testing with Presto `0.247` now and I still cannot read the table correctly.
I did some research and found out Hudi was bumped to `0.9.0` in PrestoDB `0.265` ([PR](https://github.com/prestodb/presto/commit/b52db2c4c2baa720489ae3908acd9303a41081fc), [release notes](https://prestodb.io/docs/current/release/release-0.265.html)). For Presto `<0.265`, Hudi `0.5.3` is installed.
I believe that's where the problem is. My version of Presto only has Hudi `0.5.3` and thus I cannot correctly read replacecommit files.
The Hudi docs](https://hudi.apache.org/docs/query_engine_setup#PrestoDB) do not mention anything about this issue. It only says "Presto >= 0.240 | No action needed. Hudi 0.5.3 version is a compile time dependency."
Am I correct assuming Hudi `0.5.3` cannot correctly read replacecommit files? If yes, could we maybe update the Docs to reflect this?
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-997536022
@YannByron : Alternatively you can give the exact commands you ran and BenjMaq can see how does his/her commands differ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1004675484
@codope Hive is automatically synced. It works correctly for inserts, updates, deletes. I followed [these steps](https://hudi.apache.org/docs/query_engine_setup#hive) initially for Presto and Hive and it has always worked without the need for an extra manual sync. I'm just wondering why `insert overwrite` does not work the same way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1037241618
Hi team, wondering if anyone was able to corroborate (or refute) my above statement.
Also wondering if this issue could be re-opened?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1014468802
Hi @codope , sorry for late reply...
I have tested with `master` and I'm still facing the same above issue. Presto/Hive do not recognize `*.replacecommit` files I believe and I am not able to query the inserted overwritten data using Presto/Hive. As mentioned previously, I CAN query these using Scala (`spark.read.format("hudi"`).
Were you able to test using `0.9.0`?
To be honest I'm out of option here. I've tried and tested so many things and still hitting this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
YannByron commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1004473756
@BenjMaq Does Spark-SQL work correctly also?
Can @xushiyan give some advices for Hive/Presto?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1018629404
I will let @codope follow up here. but when you query via hive, may I know what input format are you using? HiveInputFormat, or HoodieCombineHiveInputFormat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq edited a comment on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq edited a comment on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1021303523
Hi again,
Still trying to understand the issue :)
Here's the create table statement I run to create the table using `spark-sql`:
```
CREATE TABLE IF NOT EXISTS public.test_partitioned (
id bigint,
name string,
start_date string,
dt string
) USING hudi
LOCATION 's3a://<s3_bucket>/hudi_data_lake/test_partitioned'
OPTIONS (
type = 'cow',
primaryKey = 'id',
preCombineField = 'start_date'
)
PARTITIONED BY (dt);
```
and here's the DDL when I run `show create table public.test_partitioned` in Hive cli:
```
CREATE EXTERNAL TABLE `public.test_partitioned`(
`_hoodie_commit_time` string,
`_hoodie_commit_seqno` string,
`_hoodie_record_key` string,
`_hoodie_partition_path` string,
`_hoodie_file_name` string,
`id` bigint,
`name` string,
`start_date` string)
PARTITIONED BY (
`dt` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
WITH SERDEPROPERTIES (
'path'='s3a://<s3_bucket>/hudi_data_lake/test_partitioned',
'preCombineField'='start_date',
'primaryKey'='id',
'type'='cow')
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.HoodieParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
's3a://<s3_bucket>/hudi_data_lake/test_partitioned'
TBLPROPERTIES (
'last_commit_time_sync'='20220125071309',
'spark.sql.create.version'='2.4.4-20210211',
'spark.sql.sources.provider'='hudi',
'spark.sql.sources.schema.numPartCols'='1',
'spark.sql.sources.schema.numParts'='1',
'spark.sql.sources.schema.part.0'='{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"id","type":"long","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"start_date","type":"string","nullable":true,"metadata":{}},{"name":"dt","type":"string","nullable":true,"metadata":{}}]}',
'spark.sql.sources.schema.partCol.0'='dt',
'transient_lastDdlTime'='1643122431')
```
Are you maybe seeing something that looks off? I see for example
```
STORED AS INPUTFORMAT
'org.apache.hudi.hadoop.HoodieParquetInputFormat'
```
which is neither `HiveInputFormat` nor `HoodieCombineHiveInputFormat` mentioned by @nsivabalan. Is that normal? As mentioned above, I've followed the steps in https://hudi.apache.org/docs/query_engine_setup#PrestoDB to be able to read from Presto and Hive.
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-994623507
@YannByron I use the same Spark and Hudi as you and I still have this issue in `0.10.0`.
I see the parquet files being correctly created but the table still points to the old files (not the new ones created...).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
YannByron commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-997131446
@nsivabalan I failed to reproduce this. @danny0405 can you reproduce this issues?
And @BenjMaq just execute `create table`, `insert into`, and `insert overwrite` these three steps? any other commits?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] BenjMaq commented on issue #4154: [SUPPORT] INSERT OVERWRITE operation does not work when using Spark SQL
Posted by GitBox <gi...@apache.org>.
BenjMaq commented on issue #4154:
URL: https://github.com/apache/hudi/issues/4154#issuecomment-1004806239
I'm facing that issues using Hudi `0.9.0` and `0.10.0`. I'll try with the latest master.
Thank you for following up on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org