You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Raymond Xu (Jira)" <ji...@apache.org> on 2021/03/05 03:03:00 UTC
[jira] [Updated] (HUDI-1655) Support custom date format and fix
unsupported exception in DatePartitionPathSelector
[ https://issues.apache.org/jira/browse/HUDI-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymond Xu updated HUDI-1655:
-----------------------------
Summary: Support custom date format and fix unsupported exception in DatePartitionPathSelector (was: Allow custom date format in DatePartitionPathSelector)
> Support custom date format and fix unsupported exception in DatePartitionPathSelector
> -------------------------------------------------------------------------------------
>
> Key: HUDI-1655
> URL: https://issues.apache.org/jira/browse/HUDI-1655
> Project: Apache Hudi
> Issue Type: Improvement
> Components: DeltaStreamer
> Reporter: Raymond Xu
> Assignee: Raymond Xu
> Priority: Minor
> Labels: pull-request-available
> Fix For: 0.8.0
>
>
> Add a config to allow parsing custom date format in {{DatePartitionPathSelector}}. Currently it assumes date partition string in the format of {{yyyy-MM-dd}}.
>
> Also eligibleFiles.sort() throws this exception
> {quote}java.lang.UnsupportedOperationException at java.util.AbstractList.set(AbstractList.java:132) at java.util.AbstractList$ListItr.set(AbstractList.java:426) at java.util.List.sort(List.java:482) at org.apache.hudi.utilities.sources.helpers.DatePartitionPathSelector.getNextFilePathsAndMaxModificationTime(DatePartitionPathSelector.java:141) at org.apache.hudi.utilities.sources.ParquetDFSSource.fetchNextBatch(ParquetDFSSource.java:48) at org.apache.hudi.utilities.sources.RowSource.fetchNewData(RowSource.java:43) at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:75) at org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter.fetchNewDataInRowFormat(SourceFormatAdapter.java:94) at org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:338) at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:255) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:587)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)