[jira] [Resolved] (CONNECTORS-1791) Upgrade MongoDB output connector to the MongoDB Java sync driver 5.x and clean up the GridFS connector

["Piergiorgio Lucidi (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CONNECTORS-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Piergiorgio Lucidi resolved CONNECTORS-1791.
--------------------------------------------
    Resolution: Fixed

- Replaced the legacy mongo-java-driver (2.x) with mongodb-driver-sync 5.6.5
          (MongoDB 9.x compliant) in both the MongoDB output and GridFS repository connectors.
        - MongoDB connector: ported to MongoClients/MongoDatabase API, credentials via
          MongoCredential, ping-based connection check, atomic upsert (replaceOne) for indexing.
        - GridFS connector: ported to GridFSBucket API, kept support for the legacy top-level
          contentType field, mapped transient network errors to ServiceInterruption.
        - Fixed GridFS bugs: wrong version string (operator precedence / NPE on null metadata),
          ClassCastException on ACL arrays, mutation of deny ACL metadata, client closed after
          every document.
        - Replaced flapdoodle embedded MongoDB with Testcontainers (mongo:9.0.2) for the
          integration tests; added a new GridFS integration test (Docker is now required).
        - Updated Maven and Ant builds: removed the legacy driver download, added Testcontainers
          test materials, updated test-materials READMEs.
        - Removed the OpenSSL 1.1 workaround from the CI workflow.

> Upgrade MongoDB output connector to the MongoDB Java sync driver 5.x and clean up the GridFS connector
> ------------------------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-1791
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-1791
>             Project: ManifoldCF
>          Issue Type: Improvement
>          Components: Build, GridFS connector, MongoDB Output Connector
>    Affects Versions: ManifoldCF 2.31
>            Reporter: Piergiorgio Lucidi
>            Assignee: Piergiorgio Lucidi
>            Priority: Major
>             Fix For: ManifoldCF 2.32
>
>
> h2. Description
> The MongoDB output connector was built on the legacy {{mongo-java-driver}} 2.14.3. That driver is end-of-life and cannot talk to current MongoDB servers. It relies on APIs that were removed years ago ({{{}DB{}}}, {{{}DBCollection{}}}, {{{}DBTCPConnector{}}}, {{{}DBPortPool{}}}, {{{}DB.authenticate(){}}}, {{{}WriteResult{}}}, ...).
> Its integration tests used an embedded MongoDB 4.0 started by flapdoodle. That binary needs OpenSSL 1.1, which modern Linux distributions no longer ship, and it has to be forced to x86_64 on Apple Silicon. CI therefore needed a workaround that installed {{libssl1.1}} on the runner.
> This issue moves the MongoDB output connector to the current {{mongodb-driver-sync}} 5.x API (compatible with MongoDB 9.x). It also runs the integration tests against a real MongoDB 9.0.2 instance via Testcontainers and does some code cleanup in the GridFS repository connector.
> h2. Changes
> h3. MongoDB output connector ({{{}connectors/mongodb{}}})
>  * *Driver upgrade:* {{org.mongodb:mongo-java-driver:2.14.3}} → {{org.mongodb:mongodb-driver-sync:5.6.5}} (with {{{}mongodb-driver-core{}}}, {{{}bson{}}}, {{{}bson-record-codec{}}}).
>  ** Maven: the version comes from the Spring Boot BOM imported in the root {{{}pom.xml{}}}.
>  ** Ant: new {{mongodb-driver-sync.version}} property and matching {{download-via-maven}} calls in the root {{{}build.xml{}}}.
>  * *{{MongodbOutputConnector}} moved to the new API:*
>  ** {{{}MongoClient{}}}/{{{}DB{}}} → {{MongoClients.create(MongoClientSettings)}} / {{{}MongoDatabase{}}}. Host and port default to {{{}localhost:27017{}}}; four duplicated connection branches are now one.
>  ** Authentication uses {{MongoCredential.createCredential(username, database, password)}} instead of the removed {{{}DB.authenticate(){}}}.
>  ** {{isConnected()}} and the connection check thread send a {{ping}} command instead of probing the removed {{{}DBTCPConnector{}}}/{{{}DBPort{}}} internals.
>  ** {{disconnect}} now calls {{client.close()}} to release the connection pool.
>  ** Indexing uses one atomic {{{}replaceOne(Filters.eq("documentURI", ...), doc, new ReplaceOptions().upsert(true)){}}}. This replaces the old find + count + insert/update sequence, which was not atomic and was expensive. The result is read from {{{}UpdateResult{}}}.
>  ** Deletion uses {{deleteOne(...)}} / {{{}DeleteResult.getDeletedCount(){}}}.
>  ** Removed {{{}WriteConcern.SAFE{}}}, because acknowledged writes are the default in the modern driver.
>  ** Multi-valued metadata ({{{}String[]{}}}/{{{}Date[]{}}}) is wrapped in {{List}} so the BSON codec can encode it.
>  * *Integration tests:*
>  ** Replaced flapdoodle embedded MongoDB 4.0 with {{org.testcontainers:testcontainers-mongodb}} running {{{}mongo:9.0.2{}}}.
>  ** {{BaseITSanityTestUtils}} host and port are no longer {{{}final{}}}. They are set at runtime from the container's mapped port.
>  ** {{APISanityHSQLDBIT}} uses the new driver API ({{{}runCommand{}}}, {{{}insertOne{}}}, {{{}countDocuments{}}}, {{{}drop{}}}). It now closes the test client, and teardown no longer fails if the user was never created.
>  * *Ant build ({{{}connectors/mongodb/build.xml{}}}):* rewritten. Test materials are now the Testcontainers 2.0.5 stack (docker-java 3.7.1, duct-tape, JNA, commons-compress, jackson-annotations, jetbrains annotations) instead of the flapdoodle/jgrapht/jheaps jars. Test materials go first on the test classpath so their newer transitive jars take precedence.
> h3. GridFS repository connector ({{{}connectors/gridfs{}}})
>  * {{org.apache.commons.lang.StringUtils}} → {{{}org.apache.commons.lang3.StringUtils{}}}.
>  * Static constants ({{{}EXCLUDED_URL{}}}, {{{}EXCLUDED_LENGTH{}}}, {{{}EXCLUDED_MIMETYPE{}}}, {{{}EXCLUDED_DATE{}}}, {{{}BAD_URL{}}}, {{{}NULL_URL{}}}) are now accessed through {{IProcessActivity}} instead of an instance reference.
>  * Generified {{documentKnownColumns}} ({{{}HashMap<String, String>{}}}).
>  * Removed unused static imports in {{{}Messages{}}}.
> h3. CI ({{{}.github/workflows/ci.yml{}}})
>  * Removed the temporary "Install OpenSSL 1.1 runtime for embedded legacy MongoDB" step. Testcontainers does not need it.
> h2. Behavioural notes
>  * Credentials are now checked lazily by the driver, on the first operation or {{{}ping{}}}, not when the client is created. Bad credentials appear in the connection check instead of at session creation.
>  * {{documentURI}} matching is unchanged. Re-indexing a document still replaces the existing document.
> h2. Requirements
>  * *Docker is now required* to run the MongoDB connector integration tests ({{{}mvn verify{}}} / {{{}ant test{}}}). GitHub-hosted Ubuntu runners include Docker by default.
> h2. Out of scope / follow-ups
>  * The *GridFS connector still uses the legacy driver* ({{{}mongo-java-driver{}}} 2.11.3 in Maven, 2.14.3 in Ant, through the shared {{mongodb.version}} property). Moving it to {{mongodb-driver-sync}} + {{com.mongodb.client.gridfs.GridFSBucket}} should be tracked in a separate issue. After that, the legacy driver download can be removed from the root {{{}build.xml{}}}.
> h2. Testing
>  * {{mvn clean verify -pl connectors/mongodb,connectors/gridfs -am}}
>  * {{ant make-core-deps make-deps build test}} (CI workflow)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)