You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2021/06/12 14:04:00 UTC
[jira] [Updated] (HUDI-2001) NoSuchMethodError ... PartitionedFile
in MergeOnReadSnapshotRelation
[ https://issues.apache.org/jira/browse/HUDI-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
sivabalan narayanan updated HUDI-2001:
--------------------------------------
Description:
In latest master, I was trying out hudi test suite job and ran into NoSuchMethodError. My compiled scala is 2.11 with spark 2.4.4. Run time spark version is 2.4.7. I tried in EMR. When I do snapshot read with an MOR table, running into this exception.
Stacktrace: [https://gist.github.com/nsivabalan/9a276c3c0dc8a2ac93310fedccaa2fea]
Specifically, this happens when ValidateDatasetNode is being used.
```
// read from hudi and remove meta columns.
Dataset<Row> hudiDf = session.read().format("hudi").load(hudiPath);
Dataset<Row> trimmedDf = hudiDf.drop(HoodieRecord.COMMIT_TIME_METADATA_FIELD).drop(HoodieRecord.COMMIT_SEQNO_METADATA_FIELD).drop(HoodieRecord.RECORD_KEY_METADATA_FIELD)
.drop(HoodieRecord.PARTITION_PATH_METADATA_FIELD).drop(HoodieRecord.FILENAME_METADATA_FIELD);
```
was:
In latest master, I was trying out hudi test suite job and ran into NoSuchMethodError. My compiled scala is 2.11 with spark 2.4.4. Run time spark version is 2.4.7. I tried in EMR.
Stacktrace: [https://gist.github.com/nsivabalan/9a276c3c0dc8a2ac93310fedccaa2fea]
This happens when ValidateDatasetNode is being used.
```
// read from hudi and remove meta columns.
Dataset<Row> hudiDf = session.read().format("hudi").load(hudiPath);
Dataset<Row> trimmedDf = hudiDf.drop(HoodieRecord.COMMIT_TIME_METADATA_FIELD).drop(HoodieRecord.COMMIT_SEQNO_METADATA_FIELD).drop(HoodieRecord.RECORD_KEY_METADATA_FIELD)
.drop(HoodieRecord.PARTITION_PATH_METADATA_FIELD).drop(HoodieRecord.FILENAME_METADATA_FIELD);
```
> NoSuchMethodError ... PartitionedFile in MergeOnReadSnapshotRelation
> --------------------------------------------------------------------
>
> Key: HUDI-2001
> URL: https://issues.apache.org/jira/browse/HUDI-2001
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: sivabalan narayanan
> Priority: Major
>
> In latest master, I was trying out hudi test suite job and ran into NoSuchMethodError. My compiled scala is 2.11 with spark 2.4.4. Run time spark version is 2.4.7. I tried in EMR. When I do snapshot read with an MOR table, running into this exception.
> Stacktrace: [https://gist.github.com/nsivabalan/9a276c3c0dc8a2ac93310fedccaa2fea]
> Specifically, this happens when ValidateDatasetNode is being used.
> ```
> // read from hudi and remove meta columns.
> Dataset<Row> hudiDf = session.read().format("hudi").load(hudiPath);
> Dataset<Row> trimmedDf = hudiDf.drop(HoodieRecord.COMMIT_TIME_METADATA_FIELD).drop(HoodieRecord.COMMIT_SEQNO_METADATA_FIELD).drop(HoodieRecord.RECORD_KEY_METADATA_FIELD)
> .drop(HoodieRecord.PARTITION_PATH_METADATA_FIELD).drop(HoodieRecord.FILENAME_METADATA_FIELD);
> ```
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)