You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "S3tuit (via GitHub)" <gi...@apache.org> on 2025/09/16 19:28:47 UTC

[PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

S3tuit opened a new pull request, #5702:
URL: https://github.com/apache/hop/pull/5702

   New ValueMetaUuid plugin (native class java.util.UUID).
   
   - Relational dbs support via BaseDatabaseMeta.customizeValueFromSqlType
   - MongoDB, UUID support via changes to transforms/mongodb plugin
   - junit test: ValueMetaTest
   
   Closes #5624
   ------------------------
   To make clear that you license your contribution under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
   you have to acknowledge this by using the following check-box.
   
   - [ V] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
   - [V ] In any other case, please file an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "S3tuit (via GitHub)" <gi...@apache.org>.
S3tuit commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3335706232

   > I would add a simple integration test in https://github.com/apache/hop/tree/main/integration-tests/database ... but I feel like perhaps it's going to be hard to enable the extension with `CREATE EXTENSION IF NOT EXISTS "uuid-ossp";` in the container.
   
   The error you got in the table view is fixed, before I make another commit I want to show you a way we can add unit test for the UUID.
   
   ### Add extension to postgres
   I don't think we actually need `uuid-ossp` since `pgcrypto` provides `gen_random_uuid()` and it's more used nowadays. To enable an extension on Postgres we can use the `docker-entrypoint-initdb.d` inside postgres image:
   
   ```
     postgres:
     ...
       volumes:
         - ./database-startup.sql:/docker-entrypoint-initdb.d/01-extensions.sql:ro
   ```
   
   Inside database-startup.sql: `CREATE EXTENSION IF NOT EXISTS pgcrypto;`
   
   ### Test idea
   
   1. Postgres creates a table with UUID column and insert via gen_random_uuid(). (Or manual UUID to keep it the same across runs. We don't even need extensions with this.)
   2. Read back with the UUIDs, also read the same values cast to VARCHAR, then cast back to UUIDs, sort, and join.
   3. Expected a sorted pairs match
   
   If that sounds good, I can add the integration test.
   
   The only thing I don't understand is how do you manage env variable for unit-test-db? In the env file there are just 3 variables, and in the dev-env-config.json file there's no var for postgres.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "S3tuit (via GitHub)" <gi...@apache.org>.
S3tuit commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3344115689

   @mattcasters I’ve added three Mongo integration tests for UUID handling:
   
   1. insert: write 5 pairs of UUIDs into Mongo stored as UUID types in lists
   2. delete: remove 3 of those pairs
   3. verify: read back the remaining 2 rows, assert the fields are read as UUID and confirm the row count is exactly 2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3337185191

   Fantastic work @S3tuit.  Let us know if you need help adding the integration test(s). See also:  https://hop.apache.org//dev-manual/latest/integration-testing.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3345564859

   Minor change that will be required is updating the Unit tests to Junit5.
   I don't think those tests will work on a rebase of the current main branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3338377100

   You can set the 'POSTGRES_' variables yourself in an environment.  If not set these are defined in a script called `integration-tests/script/run-tests.sh`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3301686186

   @S3tuit, Thank you so much for the contribution. We will review ASAP


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva merged PR #5702:
URL: https://github.com/apache/hop/pull/5702


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3340151945

   Great work @S3tuit !  For the MongoDB improvements: could you also add a few integration tests the `mongo` project in `integration-tests` and reference the issue number in the PR descriptin.  Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3336745513

   The Postgres variables are added in the script as parameters https://github.com/apache/hop/blob/main/integration-tests/scripts/run-tests.sh


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3345945409

   As it was a minor change, I updated the class to not block this PR, I also fixed the spotless issue.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "mattcasters (via GitHub)" <gi...@apache.org>.
mattcasters commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3332988621

   When I have a UUID data type in a column in a table view or data preview window and I want to sort on some column I get:
   
   ```
   org.apache.hop.core.exception.HopValueException: 
   some_key String : I can't convert the specified value to data type : 32
   
   	at org.apache.hop.core.row.value.ValueMetaBase.convertDataUsingConversionMetaData(ValueMetaBase.java:4641)
   	at org.apache.hop.ui.core.widget.TableView.sortTable(TableView.java:1707)
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "S3tuit (via GitHub)" <gi...@apache.org>.
S3tuit commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3340040450

   @mattcasters I pushed updates that fix the UUID sort in the table view, enable the `pgcrypto` extension at container startup, and add this integration test:
   
   1. create a table with 5 UUIDs (`gen_random_uuid(`))
   2. use two Table Input, one reads the UUID column as `UUID`, the other reads the same column as `UUID::CHAR(36)`
   3. convert the string stream back to UUID
   4. join the two streams to de-duplicate
   5. write to a table and assert it has exactly 5 rows


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add UUID plugin value type + Relational/Mongo support (issue #5624) (hop)

Posted by "S3tuit (via GitHub)" <gi...@apache.org>.
S3tuit commented on PR #5702:
URL: https://github.com/apache/hop/pull/5702#issuecomment-3304156580

   > @S3tuit, Thank you so much for the contribution. We will review ASAP
   
   I made another commit because I forgot to remove the inet svg icon from the UUID valuetype plugin, hope this isn't a big problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org