[I] Add log4j-iceberg appender module for writing logs to Apache Iceberg tables [logging-log4j2]

["gsoundar (via GitHub)" <gi...@apache.org>]

gsoundar opened a new issue, #4115:
URL: https://github.com/apache/logging-log4j2/issues/4115

   ## Feature Request
   
   ### Description
   
   Add a new `log4j-iceberg` module that provides an `IcebergAppender` plugin for writing log events as Parquet-backed rows in an Apache Iceberg table. This enables structured, columnar log storage with time-travel, schema evolution, and partition pruning capabilities out of the box.
   
   ### Motivation
   
   Modern observability pipelines increasingly rely on data lake formats (Iceberg, Delta, Hudi) for log analytics due to their advantages over flat files:
   
   - **Columnar storage** (Parquet) enables efficient analytical queries over large log volumes
   - **Partition pruning** by date allows fast time-range scans without full table reads
   - **Schema evolution** means log schemas can be extended without rewriting history
   - **Time travel** enables querying historical log state at any snapshot
   - **Catalog integration** (REST, Hive, AWS Glue) provides unified metadata management
   
   Log4j already supports structured output to databases (JDBC, Cassandra, MongoDB) and message systems (Kafka, JMS). An Iceberg appender fills the gap for the data lake ecosystem.
   
   ### Proposed Implementation
   
   A new `log4j-iceberg` module with:
   
   - `IcebergAppender` — Log4j plugin (`<Iceberg>`) that buffers events and flushes them as Parquet data files
   - `IcebergManager` — Manages catalog lifecycle, table creation, buffered writes, and commit retry
   - Table partitioned by `event_date` (day granularity)
   - Schema validation on startup when loading existing tables
   - Configurable catalog properties for S3 credentials, REST auth, etc.
   - Exponential backoff retry on commit conflicts
   
   ### Configuration Example
   
   ```xml
   <Iceberg name="IcebergAppender"
            catalogName="my_catalog"
            catalogImpl="rest"
            catalogUri="http://localhost:8181"
            catalogWarehouse="s3://my-bucket/warehouse"
            tableNamespace="logs"
            tableName="app_logs"
            batchSize="1000"
            flushIntervalSeconds="30">
     <CatalogProperties>
       <Property name="s3.access-key-id">AKIA...</Property>
       <Property name="s3.secret-access-key">secret</Property>
     </CatalogProperties>
   </Iceberg>
   ```
   
   ### Dependencies
   
   - Apache Iceberg 1.10.1
   - Apache Parquet 1.16.0
   - Hadoop 3.4.1
   
   ### Related PR
   
   - #4104


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add log4j-iceberg appender module for writing logs to Apache Iceberg tables [logging-log4j2]

["vy (via GitHub)" <gi...@apache.org>]

vy closed issue #4115: Add log4j-iceberg appender module for writing logs to Apache Iceberg tables
URL: https://github.com/apache/logging-log4j2/issues/4115


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add log4j-iceberg appender module for writing logs to Apache Iceberg tables [logging-log4j2]

["vy (via GitHub)" <gi...@apache.org>]

vy commented on issue #4115:
URL: https://github.com/apache/logging-log4j2/issues/4115#issuecomment-4476520608

   @gsoundar, thanks so much for the contribution. AFAIK, there hasn't been a prior discussion for this feature. In general, we recommend external contributors to publish custom Log4j extensions externally, e.g., for this particular case, in the `apache/iceberg-log4j` repository. If you do so, we'd be more than happy to review the code, and accept a PR updating the Log4j website with a pointer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@logging.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org