You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "chl-wxp (via GitHub)" <gi...@apache.org> on 2026/06/01 05:32:44 UTC

Re: [PR] [Feature][Zeta]Add dynamic metadata provider [seatunnel]

chl-wxp commented on PR #10838:
URL: https://github.com/apache/seatunnel/pull/10838#issuecomment-4589823653

   > Re-posting this under Daniel because the previous submission from my environment was attributed incorrectly. Please treat this Daniel-authored comment as the authoritative maintainer note for the current head.
   > 
   > Thanks for the ping. I re-reviewed the latest head from scratch and traced the current metadata-datasource path again from `JettyService` into the Hazelcast store and back into connector config merge.
   > 
   > The typed-value serialization issue and the job-level override issue are fixed now, but the management write surface is still not protected by design. So the same security-boundary blocker remains on the latest head.
   > 
   > Runtime path I checked:
   > 
   > ```
   > REST management surface
   >   -> JettyService.createJettyServer()
   >       -> BasicAuthFilter is installed only when global basic auth is explicitly enabled
   >       -> DynamicMetadataDataSourceServlet is still registered on the normal REST surface
   > 
   > Write path
   >   -> DynamicMetadataDataSourceService.create/update/deleteDatasource(...)
   >       -> writes secret-bearing datasource definitions into Hazelcast IMap
   > 
   > Job consumption path
   >   -> DynamicMetadataProvider.datasourceMap(...)
   >   -> MetadataProviderManager.mergeConfig(...)
   >       -> jobs consume the stored datasource config through metadata_datasource_id
   > ```
   > 
   > Problem 1: secret-bearing metadata datasource CRUD is still exposed on the normal Jetty management surface unless operators manually enable global basic auth
   > 
   > * Location:
   >   
   >   * `seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/JettyService.java:169-175`
   >   * `seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/JettyService.java:221-261`
   >   * `seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/service/DynamicMetadataDataSourceService.java:62-224`
   >   * `docs/en/introduction/concepts/metadata-spi.md:201-325`
   > * Severity: High
   > * Raised before by others: No
   > * Why this is still a real blocker:
   >   The service manages connector properties that can contain passwords, secret keys, tokens, and similar sensitive values. Query masking is helpful for reads, but it does not protect create/update/delete. On the current head, those write APIs are still mounted on the normal Jetty REST surface, and the only protection is the optional global basic-auth switch.
   > * Risk:
   >   Any caller that can reach the management port can create, overwrite, or delete datasource definitions that later affect jobs using `metadata_datasource_id`. That is still too open for a secret-bearing admin surface.
   > * Better fix:
   >   Option A: make auth mandatory for these endpoints instead of inheriting an opt-in global setting.
   >   Option B: split read-only and write APIs and keep the write path behind a dedicated protected admin surface.
   > 
   > ### Conclusion: fix required before merge
   > Blocking items:
   > 
   > 1. Problem 1
   > 
   > Non-blocking suggestions:
   > 
   > * Once the write surface is protected, add a security-focused test or integration check that proves unauthenticated writes are rejected.
   
   Thanks for the security review. I understand the concern about protecting
     secret-bearing datasource configs.
   
     However, I'd like to discuss whether requiring mandatory auth for this specific
     endpoint is the right approach:
   
     1. **Consistency**: Other sensitive endpoints (job submission, job stopping,
        encryption) all rely on the global basic auth setting. Adding special handling
        for metadata datasource breaks this consistent security model.
   
     2. **User scenarios**: Many users deploy in trusted environments (internal VPCs,
        behind corporate firewalls) or use external authentication (Nginx, API Gateway).
        Mandatory auth would block these valid use cases.
   
     3. **Security in depth**: Authentication is just one layer. Network isolation,
        TLS, and reverse proxy auth are equally important. Users should have the
        flexibility to choose their security architecture.
   
     **Proposed alternative**:
     - Add clear documentation warning that metadata datasource API requires
       basic auth to be enabled in production environments
     - Add a startup validation that warns (not fails) if metadata is enabled
       but basic auth is disabled
   
     This maintains flexibility while still providing clear security guidance.
   
     What do you think?
     
     In addition,do we want to make basic auth mandatory for ALL sensitive endpoints, or just this one? If we're applying stricter security, it should be a broader discussion about REST API security policy, not targeted at a single feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org