You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "davsclaus (via GitHub)" <gi...@apache.org> on 2026/06/22 20:14:30 UTC
[I] Migrate DDB integration tests away from LocalStack [camel-kamelets]
davsclaus opened a new issue, #2872:
URL: https://github.com/apache/camel-kamelets/issues/2872
The DDB integration tests (`tests/camel-kamelets-itest/src/test/resources/aws/ddb/`) have been disabled because the UpdateItem and DeleteItem operations fail against LocalStack.
Unit tests in `camel-aws2-ddb` confirm the transformer + producer pipeline works correctly (PutItem, UpdateItem, DeleteItem all produce correct DDB requests). The failures appear to be LocalStack-specific — likely the deprecated `attributeUpdates()` API used by `UpdateItemCommand` not being properly supported by LocalStack.
The tests have been disabled by renaming `*.citrus.it.yaml` → `*.citrus.it.yaml.disabled`.
### TODO
- Migrate these tests to use the new LocalStack alternative (e.g. Floki or equivalent)
- Fix the delete-item test assertion (currently expects item to still exist after delete)
- Re-enable all 4 tests: put-item, update-item, delete-item, source
### Affected files
- `aws-ddb-sink-put-item.citrus.it.yaml.disabled`
- `aws-ddb-sink-update-item.citrus.it.yaml.disabled`
- `aws-ddb-sink-delete-item.citrus.it.yaml.disabled`
- `aws-ddb-source.citrus.it.yaml.disabled`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Migrate DDB integration tests away from LocalStack [camel-kamelets]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #2872:
URL: https://github.com/apache/camel-kamelets/issues/2872#issuecomment-5812380737
Took step 1. **`DeleteItem` does not take effect, and `UpdateItem` does.**
I set the post-delete expectation in `aws-ddb-sink-delete-item.citrus.it.yaml` to `[]` — the table should be empty once the item is deleted — and ran `AwsIT`:
```
✔ SUCCESS (29346ms) aws-ddb-sink-put-item-test
✔ SUCCESS (44427ms) aws-ddb-sink-update-item-test
✘ FAILED (60438ms) aws-ddb-sink-delete-item-test
✔ SUCCESS (77008ms) aws-ddb-source-test
... every other AWS test green
```
The delete verification retried to exhaustion with `AWS DDB item verification failed` — the item is still in the table after the sink has run.
That narrows this issue in two ways.
**`UpdateItem` is no longer a problem.** This issue records both `UpdateItem` and `DeleteItem` failing against LocalStack. On Floci, `UpdateItem` passes. Only the delete is left.
**The current test asserts the broken state.** After running the sink with `DeleteItem`, it reuses the expectation from *before* the delete:
```yaml
# Verify Kamelet sink
- createVariables:
variables:
- name: "aws.ddb.items"
value: "[[id:AttributeValue(N=${aws.ddb.item.id}), ...]]"
```
So it asserts the item is still present. It passes because the delete does nothing, and it will start failing the day the delete begins working. That is worth fixing regardless of the underlying cause, and it is why I have not pushed the one-line assertion change on its own — correcting it turns a green test red without fixing the behaviour.
## What is still unanswered
Whether Floci does not implement `DeleteItem`, or something in the producer path drops it. That needs the direct `aws2-ddb` to Floci test with no Kamelet in between, which I still have not run. Given `UpdateItem` works through the same Kamelet and the same emulator, the emulator looks the more likely of the two, but that is inference rather than evidence.
## Suggested next step
Rather than leaving an assertion that documents a bug as correct, replace it with an explicit, commented acknowledgement that the delete is not verified pending this issue — so the test neither passes for the wrong reason nor breaks when the behaviour is fixed. Happy to do that, or to run the direct producer test first if you would rather pin the cause before touching the test.
---
_Claude Code on behalf of Andrea Cosentino_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Migrate DDB integration tests away from LocalStack [camel-kamelets]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd closed issue #2872: Migrate DDB integration tests away from LocalStack
URL: https://github.com/apache/camel-kamelets/issues/2872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Migrate DDB integration tests away from LocalStack [camel-kamelets]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #2872:
URL: https://github.com/apache/camel-kamelets/issues/2872#issuecomment-5475459943
Worked through this one. Two of the three TODOs are already done; the third is not just an assertion fix, and correcting it uncovers a real problem. No PR yet, because the honest fix is not a one-liner.
## Already done
**Migrated off LocalStack.** `citrus-application.properties` now runs Floci:
```properties
# Use Floci as the AWS emulator (drop-in replacement for LocalStack)
citrus.testcontainers.localstack.image.name=floci/floci
```
**All four tests are re-enabled.** No `.disabled` files remain under `aws/ddb/` — `put-item`, `update-item`, `delete-item` and `source` are all live, and they pass:
```
Tests run: 15, Failures: 0, Errors: 0, Skipped: 0 -- in AwsIT
PASS awsDdb()[1] PASS awsDdb()[2] PASS awsDdb()[3] PASS awsDdb()[4]
```
(The `.disabled` files still in the tree are cassandra, redis, mqtt5 and nats — unrelated to this issue.)
## The delete-item assertion is worse than "wrong"
The TODO is right that the assertion is broken. The test seeds an item directly, verifies it is present, runs the Kamelet with `operation=DeleteItem`, and then asserts:
```yaml
# Verify Kamelet sink
- createVariables:
variables:
- name: "aws.ddb.items"
value: "[[id:AttributeValue(N=${aws.ddb.item.id}), title:..., year:...]]"
```
That is the **pre-delete** state. Because the surrounding `repeatOnError` succeeds as soon as the assertion holds, it passes on the first attempt against the item that has not been deleted yet, and never observes the delete at all. It is a false green, not merely a wrong expectation.
For contrast, `update-item` gets this right — its final assertion expects the *new* title and the added `directors` attribute, so it genuinely verifies the update.
## Correcting it exposes a functional failure
I changed the post-delete expectation to `[]` and ran `AwsIT` against Floci. It fails:
```
Caught exception 'AWS DDB item verification failed' - performing retry #1
... through retry #20
Tests run: 15, Failures: 0, Errors: 1 -- in AwsIT
```
Twenty retries over roughly twenty seconds and the item is still there. No DDB exception is logged in that window, and the seeded item verifies as present immediately beforehand, so the run is healthy right up to the delete.
The Kamelet side looks correct. Probing `aws-ddb-sink` with `camel run`, replacing the terminal endpoint with a log, `DeleteItem` produces exactly the headers it should:
```
CamelAwsDdbKey={id=AttributeValue(N=1)}, CamelAwsDdbOperation=DeleteItem, CamelAwsDdbReturnValues=ALL_OLD
```
and the body shape the test sends (`{"id": 3757}`, no `key` wrapper) is handled correctly by the transformer — with no `key` node it treats the whole body as the key.
So the remaining question is whether Floci actually honours `DeleteItem` (this issue notes the original LocalStack failure was on exactly `UpdateItem` and `DeleteItem`), or whether something in the producer path drops it. That needs a direct `aws2-ddb` to Floci test with no Kamelet in between, which I have not done.
## Why no PR
Fixing the assertion alone turns a passing test red without fixing the behaviour, and leaving it as it stands keeps a test that cannot fail. Neither is worth merging on its own, so I have left the tree untouched and am reporting instead.
Suggested split, if that is agreeable:
1. determine whether Floci implements `DeleteItem` (direct `aws2-ddb` test, no Kamelet)
2. depending on the answer, either fix the delete path or document the emulator gap and skip the assertion explicitly rather than inverting it
3. then correct the assertion, so the test is green because the delete works
Happy to take step 1 if you want it pursued.
Related: #2941 is a genuine `aws-ddb-sink` operation bug I fixed in #2992 — the inbound `dataTypes` default made the Kamelet's `operation` property ineffective. It is not the cause here, since this test carries the workaround that sets the operation exchange property in the calling route, but it is worth landing before anyone re-tests this.
---
_Claude Code on behalf of Andrea Cosentino_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Migrate DDB integration tests away from LocalStack [camel-kamelets]
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on issue #2872:
URL: https://github.com/apache/camel-kamelets/issues/2872#issuecomment-5830400149
Ran the direct producer test. **My last two comments were wrong: Floci is fine, `aws-ddb-sink` is fine, and the delete never happened because the test's JSON body never reached Camel as JSON.** PR: #3063.
## What the integration was actually doing
The Citrus log only shows the verification retrying to exhaustion, which is what led me to blame the delete. The integration's own dump (`citrus.camel.cli.dump.integration.output=true`, at `.citrus-jbang/i-aws-ddb-sink-route-output.txt`) shows the route dying one step *before* DynamoDB:
```
[JVM System Property] aws.ddb.json.data = {id: 2171}
...
Failed delivery for (MessageId: ...) at aws-ddb-sink-1[transformDataType1] aws-ddb-sink.kamelet.yaml:153
org.apache.camel.CamelExecutionException: Failed to get mandatory Json node from message body
at Ddb2JsonDataTypeTransformer.getBodyAsJsonNode(Ddb2JsonDataTypeTransformer.java:160)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('i' (code 105)):
was expecting double-quote to start field name
```
`{id: 2171}` — the quotes are gone. Citrus hands each system property to Camel JBang as `-Dkey="value"`; `JBangSupport` formats it literally as `-D%s="%s"`. A value containing bare double quotes collides with that wrapper and arrives unquoted.
This test was the only one under `aws/ddb/` that wrote its body with bare quotes:
| test | body as written | reaches Camel as |
|---|---|---|
| `put-item` | `{ \"id\": ..., \"year\": ..., \"title\": \"...\" }` | valid JSON |
| `update-item` | `{ \"key\": {\"id\": ...}, \"item\": {...} }` | valid JSON |
| `delete-item` | `{"id": ...}` | `{id: ...}` — **not JSON** |
So no DynamoDB request was issued at all, the seeded item survived, and the assertion expected exactly the untouched item. Green for precisely the wrong reason.
I should have read the integration dump before concluding anything about the emulator — the shared route filename means all three sink tests overwrite the same dump file, so it only survives if the delete test runs last or alone.
## Retracting the two claims
**"Floci does not implement `DeleteItem`."** It does. Standalone `floci/floci:latest`, no Camel involved:
```
$ aws dynamodb delete-item --table-name movies --key '{"id":{"N":"1234"}}' --return-values ALL_OLD
{"Attributes": {"id": {"N": "1234"}, "year": {"N": "1985"}, "title": {"S": "Back to the future"}}}
$ aws dynamodb scan --table-name movies
{"Items": [], "Count": 0, ...}
```
**"Something in the producer path drops it."** It does not. The same route the test runs, via `camel run` against standalone Floci, with the working-tree `kamelets/`:
```
SENDING {"id": 1234}
AFTER op=DeleteItem key={id=AttributeValue(N=1234)} attrs={id=AttributeValue(N=1234), year=AttributeValue(N=1985), title=AttributeValue(S=Back to the future)}
```
Table empty afterwards. Same on Camel 4.22.0 and 4.23.0-SNAPSHOT, and against real LocalStack 4.14.0 as well.
One aside worth recording, since it cost me a run: pointing `--local-kamelet-dir` at the repository root instead of `kamelets/` silently falls back to the catalog bundled with the Camel CLI. On 4.22.0 that catalog still carries the `dataTypes.in.default: json` bug from #2941, so the Kamelet turns every message into a `PutItem`. That is a stale-catalog artefact, not current behaviour.
## The fix
Both lines, in #3063:
```diff
- {"id": ${aws.ddb.item.id}}
+ { \"id\": ${aws.ddb.item.id} }
```
```diff
- value: "[[id:AttributeValue(N=${aws.ddb.item.id}), title:..., year:...]]"
+ value: "[]"
```
`AwsIT#awsDdb` with the fix:
```
✔ TEST SUCCESS: aws-ddb-source-test
✔ TEST SUCCESS: aws-ddb-sink-put-item-test
✔ TEST SUCCESS: aws-ddb-sink-delete-item-test
✔ TEST SUCCESS: aws-ddb-sink-update-item-test
EXIT=0
```
The delete verification satisfies on its first attempt now, and the dump shows `aws.ddb.json.data = { "id": 7141 }` arriving intact.
That closes the last of this issue's three TODOs: migrated off LocalStack, all four tests re-enabled, and the delete-item assertion now correct and meaningful.
---
_Claude Code on behalf of Andrea Cosentino_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org