You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/27 11:29:40 UTC
[GitHub] [hudi] jtchen-study opened a new issue, #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
jtchen-study opened a new issue, #6224:
URL: https://github.com/apache/hudi/issues/6224
Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant:[20220727185451835__commit__COMPLETED]
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.<init>(BaseRollbackActionExecutor.java:93)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.<init>(BaseRollbackActionExecutor.java:73)
at org.apache.hudi.table.action.rollback.CopyOnWriteRollbackActionExecutor.<init>(CopyOnWriteRollbackActionExecutor.java:48)
at org.apache.hudi.table.HoodieJavaCopyOnWriteTable.rollback(HoodieJavaCopyOnWriteTable.java:236)
at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:767)
... 11 more
I use java client hudi-0.11.0 version， when data is large, hudi client produces the above error, could anyone help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6224:
URL: https://github.com/apache/hudi/issues/6224#issuecomment-1216127634
closing the issue as we have the fix with 0.12.
thanks for reaching out. appreciate you keeping the community buzzing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
codope commented on issue #6224:
URL: https://github.com/apache/hudi/issues/6224#issuecomment-1206097133
@jtchen-study Ideally, rollback is triggered only for failed writes. As such fallback to listing-based rollback should be safe but we need to understand how rollback got triggered for completed instant. Can you describe the sequence of events that happened and also the setup. Steps to reproduce would be very helpful. Is this a single-writer or multi-writer scenario?
I've create HUDI-4550 to track the investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
URL: https://github.com/apache/hudi/issues/6224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6224:
URL: https://github.com/apache/hudi/issues/6224#issuecomment-1209717692
We have put in a fix for this https://github.com/apache/hudi/pull/6313
Can you give that a try and let us know if it resolves the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtchen-study commented on issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
jtchen-study commented on issue #6224:
URL: https://github.com/apache/hudi/issues/6224#issuecomment-1199005602
@yihua i am trying to develop a hudi writer plugin into DATAX， but i found that java client have some performance issue, besides the above one issue, do you have any suggession about perfermance enhance? as i know that java client does not support multi thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] tangqigithub commented on issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
tangqigithub commented on issue #6224:
URL: https://github.com/apache/hudi/issues/6224#issuecomment-1253137990
> The validation is legitimate, i.e. for a completed instant the marker dir is removed, so rollback cannot be done based on markers. Just to go past this issue, you can disable `hoodie.rollback.using.markers` (https://hudi.apache.org/docs/configurations#hoodierollbackusingmarkers). To fix this properly, I would suggest that Hudi either falls back to `ListingBasedRollbackStrategy` in such cases, or fails fast (at the time of scheduling rollback itself so that users can disable rollback using markers before rollback plan is written). @yihua wdyt?
Does disabling 'hoodie.rollback.using.markers' make rollback inefficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtchen-study commented on issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
jtchen-study commented on issue #6224:
URL: https://github.com/apache/hudi/issues/6224#issuecomment-1197915324
@yihua @danny0405 could help ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #6224:
URL: https://github.com/apache/hudi/issues/6224#issuecomment-1198736769
@jtchen-study can you share the Hudi configs and the setup you use for Java client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtchen-study commented on issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
jtchen-study commented on issue #6224:
URL: https://github.com/apache/hudi/issues/6224#issuecomment-1198993882
@yihua yes , thanks for your reply
` public static HoodieWriteConfig buildCfg(String tablePath, String tableSchema, String tableName) {
return HoodieWriteConfig.newBuilder().withPath(tablePath)
.withSchema(tableSchema)
.withFileSystemViewConfig(FileSystemViewStorageConfig.newBuilder()
.withStorageType(FileSystemViewStorageType.MEMORY)
.build())
.withEmbeddedTimelineServerEnabled(false)
.withAutoCommit(true)
.withWriteConcurrencyMode(WriteConcurrencyMode.SINGLE_WRITER)
.withMarkersType(MarkerType.DIRECT.toString())
.withEngineType(EngineType.JAVA)
.forTable(tableName)
.withIndexConfig(HoodieIndexConfig.newBuilder()
.withIndexType(HoodieIndex.IndexType.BLOOM).build())
.withCompactionConfig(HoodieCompactionConfig.newBuilder()
.withAutoClean(false)
.withAutoArchive(false)
.build())
.build();
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
codope commented on issue #6224:
URL: https://github.com/apache/hudi/issues/6224#issuecomment-1204851469
The validation is legitimate, i.e. for a completed instant the marker dir is removed, so rollback cannot be done based on markers.
Just to go past this issue, you can disable `hoodie.rollback.using.markers` (https://hudi.apache.org/docs/configurations#hoodierollbackusingmarkers). To fix this properly, I would suggest that Hudi either falls back to `ListingBasedRollbackStrategy` in such cases, or fails fast (at the time of scheduling rollback itself so that users can disable rollback using markers before rollback plan is written). @yihua wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6224: [SUPPORT] Caused by: java.lang.IllegalArgumentException: Cannot use marker based rollback strategy on completed instant
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6224:
URL: https://github.com/apache/hudi/issues/6224#issuecomment-1206763497
from the configs, looks like a single writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org