You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "chenbodeng719 (via GitHub)" <gi...@apache.org> on 2023/03/23 17:49:48 UTC
[GitHub] [hudi] chenbodeng719 opened a new issue, #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that there is only log file without parquet?
chenbodeng719 opened a new issue, #8279:
URL: https://github.com/apache/hudi/issues/8279
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
It seems that the pagee is 404.
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
I use flink to bulk insert a mor table with bucket index. But it seems that there is only log file without parquet. Is it normal? If it's normal, the offline compaction seems not working.
<img width="548" alt="image" src="https://user-images.githubusercontent.com/104059106/227296181-a69ddaeb-61d7-45b1-bc20-459a7d83bb1e.png">
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493552591
I see this pr https://github.com/apache/hudi/pull/8308. It seems that this feature hasn't been merged? So 0.13.0 doesn't support this feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493538014
I'm sure that they are unique. I test upsert in spark. It's done as expect. But it's very slow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1492970849
I use below conf to test bulk insert. There is only one parquet. Did I miss something? I expect 5 parquet. My dataset is about 120GB.
```
CREATE TABLE hbase2hudi_sink(
uid STRING PRIMARY KEY NOT ENFORCED,
oridata STRING,
update_time TIMESTAMP_LTZ(3)
) WITH (
'table.type' = 'MERGE_ON_READ',
'connector' = 'hudi',
'path' = '%s',
'write.operation' = 'bulk_insert',
'precombine.field' = 'update_time',
'write.tasks' = '2',
'index.type' = 'BUCKET',
'hoodie.bucket.index.hash.field' = 'uid',
'hoodie.bucket.index.num.buckets' = '5'
)
```
<img width="835" alt="image" src="https://user-images.githubusercontent.com/104059106/229291867-c6c4f9fa-1183-4adb-838b-c72684868b6f.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493685761
> I see this pr #8308. It seems that this feature hasn't been merged? So 0.13.0 doesn't support this feature?
Spark support for bulk_insert with bucket index is on-going, Flink already supports this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493725310
It's weird. My uid keys is like below. It seems that bucket hash function think they are 1.
```
|tiq_fb3c7524-206c-4cef-a87f-4e6379190f38:htmtalent |
|tiq_fb3c753a-11a6-4b62-8a0e-54f0df7a980a:htmtalent |
|tiq_fb3c756b-c3d4-4ab2-b7a8-38e5540619b2:htmtalent |
|tiq_fb3c76f0-2bb8-43ca-88ff-190c6528f585:htmtalent
```
When I add uid key like 1,2,3,4...， there comes out multi parquet as expected. Do I miss something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that there is only log file without parquet?
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1488618172
At first, I want to bulk insert hudi to load all data fastly with "write.tasks=256". Then I stop the bulk insert and continue to upsert with "write.tasks=256" to catch up with the msg from kafka. After I catch up with the delay msg, I want to reduce the consumer cluster to change write.tasks to 50. It turns out that this will cause data loss. Is this as expected? Am I doing wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493546337
I test upsert with bucket index in spark. It's ok. But bulk insert with bucket index seems not as expected as I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1492824086
You do not declare the index type as bucket while doing the bulk_insert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493753828
Is there some key len check? It's hard to believe there is too many conflicts....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1492884944
> You do not declare the index type as bucket while doing the bulk_insert.
So do you mean I should change my bulk insert conf like below
```
CREATE TABLE xxxx2hudi_sink(
uid STRING PRIMARY KEY NOT ENFORCED,
oridata STRING,
update_time TIMESTAMP_LTZ(3)
) WITH (
'table.type' = 'MERGE_ON_READ',
'connector' = 'hudi',
'path' = '%s',
'write.operation' = 'bulk_insert',
'precombine.field' = 'update_time',
'write.tasks' = '256',
'hoodie.index.type' = 'BUCKET',
'hoodie.bucket.index.hash.field' = 'uid',
'hoodie.bucket.index.num.buckets' = '256'
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that there is only log file without parquet?
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1482213810
For `BULK_INSERT` operation with bucket index, the writer always generates parquets directly, can you share your job configurations? It seems the `BULK_INSERT` dees not really take effect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that there is only log file without parquet?
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1488603601
> > your flink app can't start with checkpoint
>
> Are there any exceptions thrown out here?
Nope. But I think it is not a right way to solve the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493532766
How many distinct uid do you have in your dataset?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 closed issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 closed issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
URL: https://github.com/apache/hudi/issues/8279
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that there is only log file without parquet?
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1486132374
After that, there is another issue that your flink app can't start with checkpoint. You can try to delete the .hoodie diretory (except .hoodie/hoodie.properties ). Then start with checkpoint , it works for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493539406
> I'm sure that they are unique. I test upsert in spark. It's done as expect. But it's very slow.
> How many distinct uid do you have in your dataset?
I'm sure that they are unique. And we have 900,000,000 uid with a string field which has user info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493692921
> > I see this pr #8308. It seems that this feature hasn't been merged? So 0.13.0 doesn't support this feature?
>
> Spark support for bulk_insert with bucket index is on-going, Flink already supports this.
So you mean the uid in my test database is not consistent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493775969
Finally, it's the problem that the record key string can not have character ":" int it. I think there should have some reminders. It takes time to figure it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that there is only log file without parquet?
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1486126389
You are right. It turns out when you bulk insert that write.tasks would be same with hoodie.bucket.index.num.buckets. Then you have to build your bucket index during flink bootstrap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that there is only log file without parquet?
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1486241836
> your flink app can't start with checkpoint
Are there any exceptions thrown out here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1492207358
- bulk insert conf
```
CREATE TABLE hbase2hudi_sink(
uid STRING PRIMARY KEY NOT ENFORCED,
oridata STRING,
update_time TIMESTAMP_LTZ(3)
) WITH (
'table.type' = 'MERGE_ON_READ',
'connector' = 'hudi',
'path' = '%s',
'write.operation' = 'bulk_insert',
'precombine.field' = 'update_time',
'write.tasks' = '256'
)
```
- build index
```
'index.bootstrap.enabled' = 'true'
```
- restart after build index and catch up with the consumer
```
CREATE TABLE hudi_sink(
uid STRING PRIMARY KEY NOT ENFORCED,
oridata STRING,
update_time TIMESTAMP_LTZ(3)
) WITH (
'table.type' = 'MERGE_ON_READ',
'connector' = 'hudi',
'path' = '%s',
'write.operation' = 'upsert',
'precombine.field' = 'update_time',
'payload.class' = 'org.apache.hudi.common.model.OverwriteNonDefaultsWithLatestAvroPayload',
'write.tasks' = '256',
'hoodie.index.type' = 'BUCKET',
'hoodie.bucket.index.hash.field' = 'uid',
'hoodie.bucket.index.num.buckets' = '256',
'index.bootstrap.enabled' = 'false'
)
```
- reduce cluster conf
```
CREATE TABLE hudi_sink(
uid STRING PRIMARY KEY NOT ENFORCED,
oridata STRING,
update_time TIMESTAMP_LTZ(3)
) WITH (
'table.type' = 'MERGE_ON_READ',
'connector' = 'hudi',
'path' = '%s',
'write.operation' = 'upsert',
'precombine.field' = 'update_time',
'payload.class' = 'org.apache.hudi.common.model.OverwriteNonDefaultsWithLatestAvroPayload',
'write.tasks' = '50',
'hoodie.index.type' = 'BUCKET',
'hoodie.bucket.index.hash.field' = 'uid',
'hoodie.bucket.index.num.buckets' = '256',
'index.bootstrap.enabled' = 'false'
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenbodeng719 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "chenbodeng719 (via GitHub)" <gi...@apache.org>.
chenbodeng719 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493728296
> Yes, seems some hash conflicts maybe.
But we have about 1 million uid like this, all them are to hash to 1? Can't believe it.....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1493727082
Yes, seems some hash conflicts maybe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8279: [SUPPORT]I use flink to bulk insert a mor table with bucket index. But it seems that you can not change the write.tasks when you stop insert and continue upsert
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8279:
URL: https://github.com/apache/hudi/issues/8279#issuecomment-1489944316
Change parallelism should not impact the correctness, can you share more write config details? Did you also change the bucket number ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org