You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu> on 2024/10/04 11:03:58 UTC
Change in asterixdb[master]: [ASTERIXDB-3503][EXT] Deltalake format support
From Murtadha Hubail <mh...@apache.org>:
Attention is currently required from: Ian Maxon, Wail Alkowaileet, Ayush Tripathi, Ali Alsuliman.
Murtadha Hubail has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18708 )
Change subject: [ASTERIXDB-3503][EXT] Deltalake format support
......................................................................
Patch Set 15:
(5 comments)
File asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/deltalake/DeltaTableGenerator.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18708/comment/9432e549_d539e579
PS15, Line 100: String, String
When the generic type is known, don't specify it and just use <> (i.e., HashMap<>()). Some of other generics
File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18708/comment/18974222_e7fe1a96
PS15, Line 491: table_metadata_path
fix the casing
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18708/comment/f4d9a49c_f1d676cc
PS15, Line 508: DeltaLog deltaLog = DeltaLog.forTable(conf, table_metadata_path);
: Snapshot snapshot = deltaLog.snapshot();
: List<AddFile> dataFiles = snapshot.getAllFiles();
: StringBuilder builder = new StringBuilder();
: for (AddFile batchFile : dataFiles) {
: builder.append(",");
: String path = batchFile.getPath();
: builder.append(table_metadata_path).append('/').append(path);
: }
: if (builder.length() > 0) {
: builder.deleteCharAt(0);
: }
extract this to its own method
File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/aws/s3/S3Utils.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18708/comment/76396465_782a9035
PS15, Line 276: configuration.containsKey(ExternalDataConstants.TABLE_FORMAT)
: && configuration.get(ExternalDataConstants.TABLE_FORMAT).equals(ExternalDataConstants.FORMAT_DELTA))
refactor this check somewhere (e.g., isDeltaTable(config)) and use it everywhere you need to do this check
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18708/comment/59614c08_ea69a7fc
PS15, Line 278: if (!(configuration.get(ExternalDataConstants.KEY_FORMAT) == null || configuration
: .get(ExternalDataConstants.KEY_FORMAT).equals(ExternalDataConstants.FORMAT_PARQUET))) {
: throw new CompilationException(ErrorCode.INVALID_DELTA_PARAMETER);
: }
extract to something like validateDeltaProperties()
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18708
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: Iff608397aab711f324861fe83eeb428f73682912
Gerrit-Change-Number: 18708
Gerrit-PatchSet: 15
Gerrit-Owner: Ayush Tripathi <Ay...@couchbase.com>
Gerrit-Reviewer: Ali Alsuliman <al...@gmail.com>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <mh...@apache.org>
Gerrit-Reviewer: Wail Alkowaileet <wa...@gmail.com>
Gerrit-Attention: Ian Maxon <im...@apache.org>
Gerrit-Attention: Wail Alkowaileet <wa...@gmail.com>
Gerrit-Attention: Ayush Tripathi <Ay...@couchbase.com>
Gerrit-Attention: Ali Alsuliman <al...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 11:03:58 +0000
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Gerrit-MessageType: comment