Problems appending file to Athena

[Taher Koitawala <ta...@gmail.com>]

Hi All,
          I am trying to append data files to an existing iceberg table in
Athena. I wrote a Java app that uses the Iceberg apis to do this. Attached
below is a sample source code. I do see parquet files being placed on s3
however I cannot read those files in Athena. Am I missing something?

```
//Use glueCatalog and loaded table
Table table =
catalog.loadTable(TableIdentifier.of(Namespace.of("tpc_ds"),"store_sales"));

//created a file appender
FileAppender<Record> appender
=factory.newAppender(S3OutputFile.fromLocation(path,s3Client),
FileFormat.PARQUET);

//Added records to appender
appender.add(record);

//get len and close appender
long fileLen = appender.length();
appender.close();

//get metrics
appender.metrics();

//register datafiles with table
Transaction transaction = table.newTransaction();

//Append data file
AppendFiles appendFiles = transaction.newAppend();
            appendFiles.appendFile(DataFiles.builder(spec)
.withMetrics(metrics)
.withPath(location)
.withFormat(FileFormat.PARQUET)
.withRecordCount(metrics.recordCount())
.withFileSizeInBytes(fileLen).
build());

    //commit append
    appendFiles.commit();

    //commit transaction
    transaction.commitTransaction();

    --program end

```

Regards,
Taher Koitawala

Re: Problems appending file to Athena

[Taher Koitawala <ta...@gmail.com>]

I am using Glue catalog. I figured out the issue here. When i used rowDelta
transaction and FanoutWriters the issue was resolved and data was available
in Athena.

Thanks for the input Ryan

On Thu, 30 Jun, 2022, 4:29 am Ryan Blue, <bl...@tabular.io> wrote:

> Taher, what catalog are you using? You would need to use the same catalog
> that Athena is talking to.
>
> On Mon, Jun 27, 2022 at 10:55 PM Taher Koitawala <ta...@gmail.com>
> wrote:
>
>> Hi All,
>>           I am trying to append data files to an existing iceberg table
>> in Athena. I wrote a Java app that uses the Iceberg apis to do this.
>> Attached below is a sample source code. I do see parquet files being placed
>> on s3 however I cannot read those files in Athena. Am I missing something?
>>
>> ```
>> //Use glueCatalog and loaded table
>> Table table =
>> catalog.loadTable(TableIdentifier.of(Namespace.of("tpc_ds"),"store_sales"));
>>
>> //created a file appender
>> FileAppender<Record> appender
>> =factory.newAppender(S3OutputFile.fromLocation(path,s3Client),
>> FileFormat.PARQUET);
>>
>> //Added records to appender
>> appender.add(record);
>>
>> //get len and close appender
>> long fileLen = appender.length();
>> appender.close();
>>
>> //get metrics
>> appender.metrics();
>>
>> //register datafiles with table
>> Transaction transaction = table.newTransaction();
>>
>> //Append data file
>> AppendFiles appendFiles = transaction.newAppend();
>>             appendFiles.appendFile(DataFiles.builder(spec)
>> .withMetrics(metrics)
>> .withPath(location)
>> .withFormat(FileFormat.PARQUET)
>> .withRecordCount(metrics.recordCount())
>> .withFileSizeInBytes(fileLen).
>> build());
>>
>>     //commit append
>>     appendFiles.commit();
>>
>>     //commit transaction
>>     transaction.commitTransaction();
>>
>>     --program end
>>
>> ```
>>
>> Regards,
>> Taher Koitawala
>>
>
>
> --
> Ryan Blue
> Tabular
>

Re: Problems appending file to Athena

[Ryan Blue <bl...@tabular.io>]

Taher, what catalog are you using? You would need to use the same catalog
that Athena is talking to.

On Mon, Jun 27, 2022 at 10:55 PM Taher Koitawala <ta...@gmail.com> wrote:

> Hi All,
>           I am trying to append data files to an existing iceberg table in
> Athena. I wrote a Java app that uses the Iceberg apis to do this. Attached
> below is a sample source code. I do see parquet files being placed on s3
> however I cannot read those files in Athena. Am I missing something?
>
> ```
> //Use glueCatalog and loaded table
> Table table =
> catalog.loadTable(TableIdentifier.of(Namespace.of("tpc_ds"),"store_sales"));
>
> //created a file appender
> FileAppender<Record> appender
> =factory.newAppender(S3OutputFile.fromLocation(path,s3Client),
> FileFormat.PARQUET);
>
> //Added records to appender
> appender.add(record);
>
> //get len and close appender
> long fileLen = appender.length();
> appender.close();
>
> //get metrics
> appender.metrics();
>
> //register datafiles with table
> Transaction transaction = table.newTransaction();
>
> //Append data file
> AppendFiles appendFiles = transaction.newAppend();
>             appendFiles.appendFile(DataFiles.builder(spec)
> .withMetrics(metrics)
> .withPath(location)
> .withFormat(FileFormat.PARQUET)
> .withRecordCount(metrics.recordCount())
> .withFileSizeInBytes(fileLen).
> build());
>
>     //commit append
>     appendFiles.commit();
>
>     //commit transaction
>     transaction.commitTransaction();
>
>     --program end
>
> ```
>
> Regards,
> Taher Koitawala
>


-- 
Ryan Blue
Tabular