You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "krutoileshii (via GitHub)" <gi...@apache.org> on 2026/01/08 18:40:47 UTC

[I] [Feature][IMAP] Support Storing IMAP in S3 with Minio or other S3 Compatable Storage [seatunnel]

krutoileshii opened a new issue, #10308:
URL: https://github.com/apache/seatunnel/issues/10308

   ### Search before asking
   
   - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
   
   
   ### Description
   
   SeaTunnel’s IMAP source currently persists state (e.g., offsets, UID tracking, checkpoints) in HDFS. This proposal adds support for **S3-compatible object storage** as an alternative state backend.
   
   
   ### Usage Scenario
   
   We use S3 compatible storage and would like to have an ability to store imap in it as we do not have access to HDFS.
   
   ### Related issues
   
   ### Proposed Behavior
   
   * Support storing IMAP source state in:
   
     * S3
     * MinIO
     * Other S3-compatible object storage
   * Configuration aligns with existing SeaTunnel S3 settings:
   
     * Endpoint
     * Bucket
     * Path/prefix
     * Credentials
   * State semantics remain consistent with the current HDFS implementation (fault tolerance and delivery guarantees unchanged).
   
   
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Improve][Docs] Update Documentation for Storing IMAP in S3 with Minio or other S3 Compatible Storage [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #10308: [Improve][Docs] Update Documentation for Storing IMAP in S3 with Minio or other S3 Compatible Storage 
URL: https://github.com/apache/seatunnel/issues/10308


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][IMAP] Support Storing IMAP in S3 with Minio or other S3 Compatable Storage [seatunnel]

Posted by "krutoileshii (via GitHub)" <gi...@apache.org>.
krutoileshii commented on issue #10308:
URL: https://github.com/apache/seatunnel/issues/10308#issuecomment-3726100265

   I think i got this working with existing config:
   
    ```
    map:
       engine*:
         map-store:
           enabled: true
           initial-mode: EAGER
           factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory
           properties:
             type: hdfs
             namespace: /seatunnel/engine/
             clusterName: seatunnel
             storage.type: s3
             s3.bucket: s3a://your-bucket
             fs.defaultFS: s3a://your-bucket
             fs.s3a.endpoint: your-endpoint
             fs.s3a.path.style.access: "true"
             fs.s3a.access.key: access_key
             fs.s3a.secret.key: secret_key
             fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Improve][Docs] Update Documentation for Storing IMAP in S3 with Minio or other S3 Compatible Storage [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10308:
URL: https://github.com/apache/seatunnel/issues/10308#issuecomment-4956743822

   Thanks for adding the production detail.
   
   The original issue here was about whether MinIO / S3-compatible storage could be used for this path at all. The new question about metadata growth, scheduled cleanup, and cluster restart failure looks like a different operational problem, so I do not think this closed thread is the best place to track it.
   
   Could you please open a new issue for that runtime behavior and include:
   1. the exact SeaTunnel version
   2. the current map-store / MinIO-related config with sensitive values masked
   3. the startup error stack after a full cluster restart
   4. the approximate metadata size or object count when startup begins to fail
   5. whether the cleanup concern is about old abandoned job metadata, active checkpoint state, or both
   
   Those details will help us determine whether this is a missing retention mechanism, a startup-scan scalability issue, or an environment-specific configuration problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Improve][Docs] Update Documentation for Storing IMAP in S3 with Minio or other S3 Compatible Storage [seatunnel]

Posted by "nancaldw (via GitHub)" <gi...@apache.org>.
nancaldw commented on issue #10308:
URL: https://github.com/apache/seatunnel/issues/10308#issuecomment-4956719763

   When IMAP data is stored in MinIO, how can I set up scheduled cleanup? In actual production scenarios, when the metadata stored in MinIO grows too large, SeaTunnel fails to start after a complete cluster restart due to the excessive metadata size.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org