You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/05 04:29:51 UTC
[GitHub] [hudi] eric9204 opened a new issue, #6308: [SUPPORT] Spark multi writer failed，seems like clazz conflict ! ! !
eric9204 opened a new issue, #6308:
URL: https://github.com/apache/hudi/issues/6308
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1232285154
@eric9204 : just so you are aware of how conflict resolution happens w/in hudi.
If two writers are writing to same file, hudi will fail one of them and will succeed the other. But if two writers are writing to two different files, both should succeed (assuming both are concurrent writers).
For eg, lets say writer1 ingests 100 records in commit C1.
and at time t10, wirter1 updates the same 100 records and ingests to hudi. wirter2 updates the same 100 records and ingests to hudi. Hudi may not be in a position to choose the winner. and so will succeed one and will fail the other. In this situations, you will see ConflictResolutionFailed error messages.
But lets say writer1 ingests 100 records in commit C1 and writer2 ingests diff set of 100 records to a diff partition in commit C2.
at time t10, wirter1 updates records as part of C1 and ingests to hudi. writer2 updates the records as part of C2 and ingests to hudi. Both write will succeed if it happens concurrently. bcoz, there is no overlap.
Hope that clarifies why one of the writer failed in your case. you can get some additions details [here](https://hudi.apache.org/docs/concurrency_control).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1232381134
@nsivabalan I know what you mean, there is another problem.
If one of writers fail, this hoodie commit transaction was failed at hudi side, but this batch was successful at spark side. So the next micro batch will consume record from last successful commit of offset. the data of last successful micro batch which is failed at hoodie side actually may be lost 。
```
22/08/05 15:01:18 INFO HoodieStreamingSink: Ignore the exception and move on streaming as per hoodie.datasource.write.streaming.ignore.failed.batch configuration
22/08/05 15:01:18 INFO HoodieStreamingSink: Micro batch id=32 succeeded
22/08/05 15:01:18 INFO BlockManager: Removing RDD 159
22/08/05 15:01:18 INFO CheckpointFileManager: Writing atomically to hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/32 using temp file hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/.32.a829b364-c1b9-4b4b-8fae-f7d866ed76e5.tmp
22/08/05 15:01:18 INFO CheckpointFileManager: Renamed temp file hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/.32.a829b364-c1b9-4b4b-8fae-f7d866ed76e5.tmp to hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/32
```
Could the community add a retry strategy to make it successful instead of just discarding it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
codope commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1210532994
@eric9204 For the first failure, can you share the full write configs? Was there any pending/inflight compaction and clustering? Can you share the `.hoodie` directory under the base path of the table? I think the lock provider is running fine but conflict resolution failed. The timeline will help us build the sequence of events and see why the resolution failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1207706928
> > I also met the second issue, this should already be fixed in master branch , you can try it out
>
> @fengjian428 Thank you for your reply，I'll try the master branch..
The second issue was indeed fixed , thank you !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1233667672
@nsivabalan thanks! I've retested with `hoodie.datasource.write.streaming.ignore.failed.batch=false` ,the spark micro-batch indeed fail when hudi commit fail.
So, should I close this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1206536013
I think the second issue was fixed in the 0.12 branch, can you try it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1206550539
I also met the second issue, will create a pr to fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1233264138
yes. we have a config which you can flip to ensure spark streaming fails if hudi write fails.
[hoodie.datasource.write.streaming.ignore.failed.batch](https://hudi.apache.org/docs/configurations/#hoodiedatasourcewritestreamingignorefailedbatch)
default value for this config is set to true until 2 weeks back. we have flipped the default to false. if you can set this config value to false, atleast you won' see any data loss. if there are any valid errors, your pipeline should fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1211486157
> @eric9204 For the first failure, can you share the full write configs? Was there any pending/inflight compaction and clustering? Can you share the `.hoodie` directory under the base path of the table? I think the lock provider is running fine but conflict resolution failed. The timeline will help us build the sequence of events and see why the resolution failed.
```
2022-08-08 13:33 /tmp/hudi/ss_bucket_dsj/.hoodie/.aux
2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/.heartbeat
2022-08-08 13:33 /tmp/hudi/ss_bucket_dsj/.hoodie/.schema
2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/.temp
1.5 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134200577.rollback
0 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134200577.rollback.inflight
1.3 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134200577.rollback.requested
1.5 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134211686.rollback
0 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134211686.rollback.inflight
1.3 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134211686.rollback.requested
1.5 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134216453.rollback
0 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134216453.rollback.inflight
1.3 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134216453.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134916968.rollback
0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134916968.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134916968.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134917661.rollback
0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134917661.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134917661.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134918933.rollback
0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134918933.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134918933.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134919571.rollback
0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134919571.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134919571.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134920158.rollback
0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134920158.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134920158.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134921135.rollback
0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134921135.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134921135.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135002565.rollback
0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135002565.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135002565.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135003386.rollback
0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135003386.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135003386.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135013265.rollback
0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135013265.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135013265.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135022179.rollback
0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135022179.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135022179.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135033233.rollback
0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135033233.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135033233.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135036166.rollback
0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135036166.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135036166.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135037732.rollback
0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135037732.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135037732.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135043606.rollback
0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135043606.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135043606.rollback.requested
1.5 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135212947.rollback
0 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135212947.rollback.inflight
1.3 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135212947.rollback.requested
1.5 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135213720.rollback
0 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135213720.rollback.inflight
1.3 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135213720.rollback.requested
1.5 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135215256.rollback
0 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135215256.rollback.inflight
1.3 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135215256.rollback.requested
1.5 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135330116.rollback
0 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135330116.rollback.inflight
1.3 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135330116.rollback.requested
1.5 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135406717.rollback
0 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135406717.rollback.inflight
1.3 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135406717.rollback.requested
2.5 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135436042.commit
0 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135436042.commit.requested
3.0 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135436042.inflight
1.5 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135447952.clean
1.6 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135447952.clean.inflight
1.6 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135447952.clean.requested
2.5 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135459430.commit
0 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135459430.commit.requested
3.0 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135459430.inflight
1.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135509582.clean
1.6 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135509582.clean.inflight
1.6 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135509582.clean.requested
2.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135540714.commit
0 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135540714.commit.requested
3.0 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135540714.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552939.clean
1.6 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552939.clean.inflight
1.6 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552939.clean.requested
1.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552965.rollback
0 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552965.rollback.inflight
1.3 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552965.rollback.requested
1.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135624549.rollback
0 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135624549.rollback.inflight
1.3 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135624549.rollback.requested
2.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135634238.commit
0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135634238.commit.requested
3.0 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135634238.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645734.clean
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645734.clean.inflight
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645734.clean.requested
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645758.rollback
0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645758.rollback.inflight
1.3 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645758.rollback.requested
0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645810.commit.requested
3.0 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645810.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135646329.rollback
0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135646329.rollback.inflight
1.3 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135646329.rollback.requested
2.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135659559.commit
0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135659559.commit.requested
3.0 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135659559.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135711854.clean
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135711854.clean.inflight
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135711854.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135719322.commit
0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135719322.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135719322.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135730874.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135730874.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135730874.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135731862.commit
0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135731862.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135731862.inflight
0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135734515.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135734515.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135736528.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135736528.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135736528.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135746974.commit
0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135746974.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135746974.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135751508.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135751508.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135751508.clean.requested
0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135753793.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135753793.inflight
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135754854.commit
0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135754854.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135754854.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135802038.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135802038.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135802038.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135815814.commit
0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135815814.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135815814.inflight
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135819555.clean
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135819555.clean.inflight
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135819555.clean.requested
2.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135849144.commit
0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135849144.commit.requested
3.0 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135849144.inflight
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906862.clean
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906862.clean.inflight
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906862.clean.requested
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906879.rollback
0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906879.rollback.inflight
1.3 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906879.rollback.requested
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135907610.rollback
0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135907610.rollback.inflight
1.3 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135907610.rollback.requested
2.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135911957.commit
0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135911957.commit.requested
3.0 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135911957.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135920430.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135920430.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135920430.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135925137.commit
0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135925137.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135925137.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135930305.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135930305.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135930305.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135935514.commit
0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135935514.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135935514.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135945906.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135945906.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135945906.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140000922.commit
0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140000922.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140000922.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140004948.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140004948.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140004948.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140006350.commit
0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140006350.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140006350.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140010972.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140010972.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140010972.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140013016.commit
0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140013016.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140013016.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140018094.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140018094.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140018094.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140020180.commit
0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140020180.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140020180.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140028764.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140028764.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140028764.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140031809.commit
0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140031809.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140031809.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140035523.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140035523.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140035523.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140042109.commit
0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140042109.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140042109.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140046546.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140046546.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140046546.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140050105.commit
0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140050105.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140050105.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140055595.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140055595.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140055595.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140109135.commit
0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140109135.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140109135.inflight
1.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140118730.clean
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140118730.clean.inflight
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140118730.clean.requested
2.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140132300.commit
0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140132300.commit.requested
3.0 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140132300.inflight
1.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140144323.clean
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140144323.clean.inflight
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140144323.clean.requested
2.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140204915.commit
0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140204915.commit.requested
3.0 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140204915.inflight
1.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140210226.clean
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140210226.clean.inflight
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140210226.clean.requested
0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140215991.commit.requested
3.0 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140215991.inflight
2.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140223090.commit
0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140223090.commit.requested
3.0 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140223090.inflight
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140227701.clean
1.6 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140227701.clean.inflight
1.6 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140227701.clean.requested
2.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140232295.commit
0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140232295.commit.requested
3.0 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140232295.inflight
0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140238784.commit.requested
3.0 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140238784.inflight
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140246252.rollback
0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140246252.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140246252.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140247998.rollback
0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140247998.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140247998.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140253101.rollback
0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140253101.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140253101.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140255842.rollback
0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140255842.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140255842.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140303187.rollback
0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140303187.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140303187.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140305407.rollback
0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140305407.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140305407.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140306513.rollback
0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140306513.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140306513.rollback.requested
0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140308986.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140308986.rollback.requested
0 2022-08-08 13:41 /tmp/hudi/ss_bucket_dsj/.hoodie/archived
781 2022-08-08 13:33 /tmp/hudi/ss_bucket_dsj/.hoodie/hoodie.properties
```
@codope The hudi table type is cow,and there was not any config about compaction or clustering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1233809993
yes, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #6308: [SUPPORT] Spark multi writer failed ! ! !
URL: https://github.com/apache/hudi/issues/6308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] eric9204 commented on issue #6308: [SUPPORT] Spark multi writer failed ! ! !
Posted by GitBox <gi...@apache.org>.
eric9204 commented on issue #6308:
URL: https://github.com/apache/hudi/issues/6308#issuecomment-1207595582
> I also met the second issue, this should already be fixed in master branch , you can try it out
@fengjian428 Thank you for your reply，I'll try the main branch..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org