You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by gk...@apache.org on 2026/10/23 06:36:08 UTC
(iggy) branch master updated: feat(server)!: unify resource options and add per-stream keys (#4615)
This is an automated email from the ASF dual-hosted git repository.
numinnex pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iggy.git
The following commit(s) were added to refs/heads/master by this push:
new b0a951ea8 feat(server)!: unify resource options and add per-stream keys (#4615)
b0a951ea8 is described below
commit b0a951ea8e96f964499dcde67e123ba88523da8a
Author: Piotr Gankiewicz <pi...@gmail.com>
AuthorDate: Fri Oct 23 08:36:03 2026 +0200
feat(server)!: unify resource options and add per-stream keys (#4615)
Stream, topic and user settings followed separate paths. Updating one
setting meant resending the name and every other field, nothing could
return a setting to its default, and each SDK encoded options its own
way per resource and transport. Encryption shared one data key across
all streams, and a read could not tell which key a stream used.
Route every create and update through one scoped catalog on the
existing ResourceOptions map. Each key declares its scopes, kind,
storage, mutability, default, visibility and criticality, and one
admission path enforces it on every transport. UpdateResource carries
only the keys that change and explicit resets, and metadata prepare
resolves it into a per-scope internal operation so defaults are
replicated, not recomputed. Names and user status stay indexed
columns while the catalog validates them.
Give every encrypted stream its own data key, supplied by the client
or generated by the admitting node, and store it only inside an
authenticated envelope wrapped under a key derived from the config
key. Reads expose the key source and fingerprints, never key bytes.
Rotating the config key replicates rewrapped envelopes and leaves
message data untouched, and boot refuses envelopes the configured
keys cannot open.
Typed options replace raw maps in every SDK, with a raw escape hatch
for keys newer than the SDK. Unknown critical state fails closed.
Protocol 0.12.0 and snapshot format 13 move together because both
layouts change.
BREAKING CHANGE: clients and servers must run protocol 0.12.0, data
directories need snapshot format 13, and UpdateStream, UpdateTopic
and UpdateUser are replaced by UpdateResource.
---
.dockerignore | 12 +-
.github/actions/cpp-bazel/pre-merge/action.yml | 33 +
.github/actions/go/pre-merge/action.yml | 39 +
.github/actions/java-gradle/pre-merge/action.yml | 4 +
.github/actions/node-npm/pre-merge/action.yml | 30 +
.github/actions/php/pre-merge/action.yml | 11 +-
.../actions/python-maturin/pre-merge/action.yml | 41 +
.github/actions/swift/pre-merge/action.yml | 40 +
.github/actions/utils/server-start/action.yml | 17 +-
.github/workflows/coverage-baseline.yml | 8 +
.gitignore | 2 +
.typos.toml | 2 +
Cargo.lock | 179 +-
Cargo.toml | 15 +-
README.md | 6 +-
.../features/step_definitions/messaging_steps.cpp | 2 +-
.../step_definitions/resource_options_steps.cpp | 475 ++++++
bdd/cpp/features/step_definitions/world.hpp | 12 +-
bdd/cpp/scripts/entrypoint.sh | 2 +
bdd/docker-compose.coverage.yml | 6 +-
bdd/docker-compose.server.yml | 3 +
bdd/docker-compose.yml | 36 +-
bdd/go/tests/basic_messaging.go | 8 +-
bdd/go/tests/leader_redirection.go | 2 +-
bdd/go/tests/resource_options.go | 607 +++++++
bdd/go/tests/suite_test.go | 23 +
bdd/go/tests/tcp_test/stream_feature_create.go | 10 +-
bdd/go/tests/tcp_test/stream_feature_get_by_id.go | 20 +-
bdd/go/tests/tcp_test/stream_feature_update.go | 13 +-
bdd/go/tests/tcp_test/stream_steps.go | 2 +-
bdd/go/tests/tcp_test/topic_feature_create.go | 37 +-
bdd/go/tests/tcp_test/topic_feature_update.go | 37 +-
bdd/go/tests/tcp_test/topic_steps.go | 7 +-
bdd/go/tests/tcp_test/users_feature_create.go | 6 +-
bdd/go/tests/tcp_test/users_feature_password.go | 2 +-
bdd/go/tests/tcp_test/users_feature_permissions.go | 3 +-
bdd/go/tests/tcp_test/users_feature_update.go | 6 +-
bdd/go/tests/tcp_test/users_steps.go | 2 +-
.../org/apache/iggy/bdd/BasicMessagingSteps.java | 18 +-
.../org/apache/iggy/bdd/ResourceOptionsSteps.java | 415 +++++
bdd/node/Dockerfile | 2 +
bdd/php/tests/SharedFeatureParser.php | 22 +-
.../php/tests/SharedFeatureParserTest.php | 54 +-
bdd/php/tests/StreamOptionsFeatureTest.php | 249 +++
bdd/python/tests/conftest.py | 24 +-
bdd/python/tests/test_basic_messaging.py | 6 +-
bdd/python/tests/test_resource_options_patch.py | 248 +++
bdd/python/tests/test_stream_options.py | 331 ++++
bdd/python/uv.lock | 2 +-
bdd/rust/Cargo.toml | 10 +
bdd/rust/tests/common/global_context.rs | 6 +-
.../rust/tests/resource_options_patch.rs | 15 +-
bdd/rust/tests/steps/leader_redirection.rs | 2 +-
bdd/rust/tests/steps/mod.rs | 1 +
bdd/rust/tests/steps/resource_options.rs | 509 ++++++
bdd/rust/tests/steps/streams.rs | 9 +-
.../rust/tests/stream_options.rs | 15 +-
bdd/scenarios/resource_options_patch.feature | 55 +
bdd/scenarios/stream_options.feature | 74 +
core/ai/mcp/Cargo.toml | 2 +-
core/ai/mcp/src/service/mod.rs | 237 ++-
core/ai/mcp/src/service/requests.rs | 92 +-
core/bench/Cargo.toml | 2 +-
core/bench/src/benchmarks/benchmark.rs | 4 +-
core/binary_protocol/Cargo.toml | 3 +-
core/binary_protocol/src/codes.rs | 10 +-
core/binary_protocol/src/consensus/header.rs | 11 +-
core/binary_protocol/src/consensus/operation.rs | 79 +-
core/binary_protocol/src/consensus/request.rs | 29 +-
core/binary_protocol/src/dispatch.rs | 129 +-
core/binary_protocol/src/error.rs | 3 +
core/binary_protocol/src/primitives/options.rs | 25 +-
core/binary_protocol/src/requests/streams/mod.rs | 9 +-
.../src/requests/streams/rewrap_stream_keys.rs | 216 +++
.../src/requests/streams/update_stream.rs | 127 --
.../src/requests/streams/update_stream_resolved.rs | 114 ++
core/binary_protocol/src/requests/system/mod.rs | 2 +
.../src/requests/system/update_resource.rs | 476 ++++++
core/binary_protocol/src/requests/topics/mod.rs | 4 +-
.../src/requests/topics/update_topic.rs | 158 --
.../src/requests/topics/update_topic_resolved.rs | 113 ++
core/binary_protocol/src/requests/users/mod.rs | 4 +-
.../src/requests/users/update_user.rs | 228 ---
.../src/requests/users/update_user_resolved.rs | 136 ++
.../src/responses/streams/get_stream.rs | 34 +-
.../src/responses/streams/get_streams.rs | 2 +
core/binary_protocol/src/responses/streams/mod.rs | 2 -
.../src/responses/streams/stream_response.rs | 20 +-
.../src/responses/streams/update_stream.rs | 19 -
.../src/responses/system/describe_options.rs | 28 +-
core/binary_protocol/src/responses/system/mod.rs | 2 +
.../update_topic.rs => system/update_resource.rs} | 4 +-
.../src/responses/topics/get_topic.rs | 3 -
.../src/responses/topics/get_topics.rs | 3 -
core/binary_protocol/src/responses/topics/mod.rs | 2 -
.../src/responses/users/get_user.rs | 1 +
.../src/responses/users/get_users.rs | 2 +
core/binary_protocol/src/responses/users/mod.rs | 2 -
.../src/responses/users/update_user.rs | 18 -
.../src/responses/users/user_response.rs | 20 +-
core/binary_protocol/src/version.rs | 26 +-
core/cli/Cargo.toml | 2 +-
core/cli/src/args/common.rs | 20 +
core/cli/src/args/mod.rs | 8 +-
core/cli/src/args/stream.rs | 84 +-
core/cli/src/args/topic.rs | 101 +-
.../src/commands/binary_streams/create_stream.rs | 64 +-
core/cli/src/commands/binary_streams/get_stream.rs | 35 +
.../cli/src/commands/binary_streams/get_streams.rs | 15 +-
.../src/commands/binary_streams/update_stream.rs | 69 +-
core/cli/src/commands/binary_system/options.rs | 56 +-
.../cli/src/commands/binary_topics/create_topic.rs | 83 +-
core/cli/src/commands/binary_topics/get_topic.rs | 14 +-
core/cli/src/commands/binary_topics/get_topics.rs | 14 +-
.../cli/src/commands/binary_topics/update_topic.rs | 101 +-
core/cli/src/commands/binary_users/create_user.rs | 6 +-
core/cli/src/commands/binary_users/update_user.rs | 46 +-
core/cli/src/main.rs | 17 +-
core/common/Cargo.toml | 11 +-
core/common/benches/stream_encryption.rs | 103 ++
core/common/examples/resource_options_fixtures.rs | 953 +++++++++++
.../examples/resource_options_fixtures/schema.rs | 990 +++++++++++
core/common/fixtures/resource_options.json | 1619 ++++++++++++++++++
core/common/src/error/iggy_error.rs | 19 +
core/common/src/http/mod.rs | 1 +
core/common/src/http/options.rs | 706 ++++++++
core/common/src/http/streams/create_stream.rs | 7 +
core/common/src/http/streams/mod.rs | 1 -
core/common/src/http/streams/update_stream.rs | 60 -
core/common/src/http/topics/create_topic.rs | 35 +-
core/common/src/http/topics/mod.rs | 1 -
core/common/src/http/topics/update_topic.rs | 86 -
core/common/src/http/users/create_user.rs | 5 +
core/common/src/http/users/mod.rs | 1 -
core/common/src/http/users/update_user.rs | 59 -
core/common/src/lib.rs | 6 +
core/common/src/traits/binary_impls/mod.rs | 26 +-
core/common/src/traits/binary_impls/streams.rs | 37 +-
core/common/src/traits/binary_impls/topics.rs | 27 +-
core/common/src/traits/binary_impls/users.rs | 184 +-
core/common/src/traits/stream_client.rs | 18 +-
core/common/src/traits/topic_client.rs | 5 +-
core/common/src/traits/user_client.rs | 10 +-
core/common/src/types/options/catalog.rs | 1798 ++++++++++++++++++++
core/common/src/types/options/codec.rs | 273 +++
core/common/src/types/options/key.rs | 83 +
core/common/src/types/options/mod.rs | 1633 +++++++++++-------
.../common/src/types/personal_access_tokens/mod.rs | 49 +-
core/common/src/types/segment.rs | 14 +-
core/common/src/types/stream/mod.rs | 73 +-
core/common/src/types/topic/mod.rs | 52 +-
core/common/src/utils/crypto.rs | 147 +-
core/common/src/utils/mod.rs | 1 +
core/common/src/utils/stream_key.rs | 1436 ++++++++++++++++
core/common/src/wire_conversions.rs | 117 +-
core/common/tests/resource_options_fixtures.rs | 776 +++++++++
core/common/tests/stream_key_erasure.rs | 277 +++
core/configs/src/common/defaults.rs | 2 +
core/configs/src/common/displays.rs | 21 +-
core/configs/src/common/system.rs | 64 +-
core/configs/src/common/validators.rs | 116 +-
core/configs/src/server_config/validators.rs | 3 +
core/connectors/runtime/Cargo.toml | 2 +-
core/connectors/runtime/src/source.rs | 5 +-
core/connectors/sources/mqtt_source/Cargo.toml | 3 +-
core/connectors/sources/mqtt_source/README.md | 4 +-
core/connectors/sources/mqtt_source/src/driver.rs | 374 ++--
core/connectors/sources/mqtt_source/src/lib.rs | 50 +-
core/consensus/src/client_table.rs | 77 +-
core/consensus/src/observability.rs | 8 +-
core/integration/Cargo.toml | 3 +-
core/integration/src/harness/helpers.rs | 2 +
core/integration/src/harness/mod.rs | 2 +-
core/integration/src/harness/seeds.rs | 30 +-
core/integration/tests/cli/common/mod.rs | 12 +
.../test_consumer_group_create_command.rs | 5 +-
.../test_consumer_group_delete_command.rs | 5 +-
.../test_consumer_group_get_command.rs | 5 +-
.../test_consumer_group_list_command.rs | 3 +-
.../test_consumer_offset_external_group_command.rs | 5 +-
.../test_consumer_offset_get_command.rs | 2 +-
.../test_consumer_offset_set_command.rs | 4 +-
.../tests/cli/general/test_help_command.rs | 2 +-
.../tests/cli/general/test_overview_command.rs | 2 +-
.../tests/cli/message/test_message_poll_command.rs | 4 +-
.../message/test_message_poll_to_file_command.rs | 4 +-
.../tests/cli/message/test_message_send_command.rs | 2 +-
.../message/test_message_send_from_file_command.rs | 2 +-
.../cli/partition/test_partition_create_command.rs | 5 +-
.../cli/partition/test_partition_delete_command.rs | 5 +-
.../tests/cli/stream/test_stream_create_command.rs | 213 ++-
.../tests/cli/stream/test_stream_delete_command.rs | 5 +-
.../tests/cli/stream/test_stream_get_command.rs | 77 +-
.../tests/cli/stream/test_stream_help_command.rs | 2 +-
.../tests/cli/stream/test_stream_list_command.rs | 11 +-
.../tests/cli/stream/test_stream_update_command.rs | 57 +-
.../tests/cli/system/test_stats_command.rs | 9 +-
.../tests/cli/topic/test_topic_create_command.rs | 7 +-
.../tests/cli/topic/test_topic_delete_command.rs | 5 +-
.../tests/cli/topic/test_topic_get_command.rs | 16 +-
.../tests/cli/topic/test_topic_help_command.rs | 2 +-
.../tests/cli/topic/test_topic_list_command.rs | 5 +-
.../tests/cli/topic/test_topic_update_command.rs | 313 ++--
core/integration/tests/cli/user/common.rs | 6 +-
.../tests/cli/user/test_user_delete_command.rs | 9 +-
.../tests/cli/user/test_user_get_command.rs | 2 +
.../tests/cli/user/test_user_list_command.rs | 9 +-
.../tests/cli/user/test_user_name_command.rs | 9 +-
.../tests/cli/user/test_user_password_command.rs | 9 +-
.../cli/user/test_user_permissions_command.rs | 9 +-
.../tests/cli/user/test_user_status_command.rs | 9 +-
core/integration/tests/cluster/chaos_monkey.rs | 2 +-
.../tests/cluster/client_table_adversarial.rs | 2 +-
.../tests/cluster/client_table_restart.rs | 25 +-
.../tests/cluster/consumer_offset_quota.rs | 2 +-
core/integration/tests/cluster/crash_durability.rs | 17 +-
.../tests/cluster/crash_offset_reuse.rs | 2 +-
.../tests/cluster/crash_recovery_corruption.rs | 9 +-
.../tests/cluster/failover_client_continuity.rs | 2 +-
.../tests/cluster/fast_primary_rejoin.rs | 2 +-
.../tests/cluster/metadata_checkpoint_restart.rs | 2 +-
core/integration/tests/cluster/mod.rs | 2 +
.../cluster/multi_shard_partition_convergence.rs | 10 +-
.../tests/cluster/parked_frame_redispatch.rs | 5 +-
core/integration/tests/cluster/partition_dedup.rs | 3 +-
.../tests/cluster/partition_primary_routing.rs | 51 +-
.../tests/cluster/partition_state_transfer.rs | 2 +-
.../tests/cluster/register_forwarding.rs | 2 +-
.../tests/cluster/replica_read_batching.rs | 5 +-
.../tests/cluster/staggered_bootstrap.rs | 2 +-
.../integration/tests/cluster/stream_encryption.rs | 451 +++++
.../tests/cluster/stream_key_rotation.rs | 485 ++++++
.../tests/connectors/fixtures/mqtt/mod.rs | 2 +-
.../tests/connectors/fixtures/mqtt/publisher.rs | 120 +-
.../tests/connectors/mqtt/mqtt_source.rs | 2 +-
.../tests/data_integrity/storage_compat.rs | 32 +-
.../data_integrity/verify_after_server_restart.rs | 20 +-
.../verify_auto_commit_offset_replicates.rs | 5 +-
.../verify_cluster_replica_data_identical.rs | 165 +-
.../verify_consumer_group_partition_assignment.rs | 40 +-
.../verify_no_plaintext_credentials_on_disk.rs | 93 +-
core/integration/tests/mcp/mod.rs | 22 +
core/integration/tests/sdk/consumer_group.rs | 11 +-
.../tests/sdk/consumer_group_membership.rs | 30 +-
core/integration/tests/sdk/consumer_offset.rs | 15 +-
core/integration/tests/sdk/consumer_shutdown.rs | 5 +-
core/integration/tests/sdk/options.rs | 381 ++++-
core/integration/tests/sdk/producer/mod.rs | 5 +-
core/integration/tests/sdk/send_confirmation.rs | 2 +-
core/integration/tests/server/a2a_jwt/jwt_tests.rs | 16 +-
.../tests/server/cluster_view_durability_vsr.rs | 2 +-
.../tests/server/consumer_offset_quota_vsr.rs | 8 +-
.../tests/server/descriptor_exhaustion_vsr.rs | 2 +-
core/integration/tests/server/http_rbac.rs | 15 +-
.../tests/server/http_read_your_writes.rs | 2 +-
core/integration/tests/server/http_vsr.rs | 159 +-
.../tests/server/partition_view_durability_vsr.rs | 2 +-
.../tests/server/partitions_limit_vsr.rs | 2 +-
.../server/personal_access_token_cleaner_vsr.rs | 15 +-
.../tests/server/pipelined_frames_vsr.rs | 5 +-
.../integration/tests/server/poll_semantics_vsr.rs | 10 +-
.../server/scenarios/authentication_scenario.rs | 40 +-
.../concurrent_produce_consume_scenario.rs | 5 +-
.../tests/server/scenarios/concurrent_scenario.rs | 36 +-
...umer_group_auto_commit_reconnection_scenario.rs | 5 +-
...onsumer_group_duplicate_name_create_scenario.rs | 6 +-
.../scenarios/consumer_group_join_scenario.rs | 6 +-
...er_group_new_messages_after_restart_scenario.rs | 5 +-
.../consumer_group_offset_cleanup_scenario.rs | 5 +-
...h_multiple_clients_polling_messages_scenario.rs | 5 +-
...with_single_client_polling_messages_scenario.rs | 5 +-
.../consumer_timestamp_polling_scenario.rs | 5 +-
.../server/scenarios/delete_segments_scenario.rs | 20 +-
.../scenarios/delete_stats_rollback_scenario.rs | 10 +-
.../tests/server/scenarios/encryption_scenario.rs | 598 ++++++-
.../scenarios/external_group_offsets_scenario.rs | 6 +-
.../scenarios/invalid_consumer_offset_scenario.rs | 5 +-
.../server/scenarios/log_rotation_scenario.rs | 2 +-
.../server/scenarios/message_cleanup_scenario.rs | 37 +-
.../server/scenarios/message_headers_scenario.rs | 5 +-
.../server/scenarios/message_size_scenario.rs | 5 +-
.../tests/server/scenarios/offset_scenario.rs | 5 +-
.../tests/server/scenarios/permissions_scenario.rs | 84 +-
.../scenarios/read_during_persistence_scenario.rs | 5 +-
.../scenarios/reconnect_after_restart_scenario.rs | 21 +-
.../scenarios/restart_offset_skip_scenario.rs | 5 +-
.../scenarios/segment_rotation_race_scenario.rs | 5 +-
.../scenarios/single_message_per_batch_scenario.rs | 5 +-
.../stale_client_consumer_group_scenario.rs | 5 +-
.../scenarios/stream_size_validation_scenario.rs | 5 +-
.../tests/server/scenarios/system_scenario.rs | 57 +-
.../tests/server/scenarios/tcp_tls_scenario.rs | 5 +-
.../tests/server/scenarios/timestamp_scenario.rs | 5 +-
.../tests/server/scenarios/user_scenario.rs | 26 +-
.../server/scenarios/websocket_tls_scenario.rs | 5 +-
.../tests/server/topic_admission_vsr.rs | 38 +-
core/integration/tests/server/topic_metrics_vsr.rs | 11 +-
core/metadata/src/impls/metadata.rs | 770 ++++++++-
core/metadata/src/impls/recovery.rs | 11 +-
core/metadata/src/stm/authz.rs | 105 +-
core/metadata/src/stm/lifecycle.rs | 72 +-
core/metadata/src/stm/mod.rs | 2 +
core/metadata/src/stm/result.rs | 55 +-
core/metadata/src/stm/snapshot.rs | 70 +-
core/metadata/src/stm/stream.rs | 1185 ++++++++++---
core/metadata/src/stm/user.rs | 272 ++-
core/partitions/src/iggy_partition.rs | 86 +-
core/partitions/src/iggy_partitions.rs | 82 +-
core/partitions/src/lib.rs | 2 +
core/partitions/src/segment.rs | 10 +-
core/partitions/src/segment_recovery.rs | 3 +-
core/partitions/src/stream_keyring.rs | 178 ++
core/partitions/src/types.rs | 14 +-
core/partitions/tests/consumer_offset_history.rs | 2 +-
core/sdk/Cargo.toml | 3 +-
core/sdk/README.md | 1 +
.../src/client_wrappers/binary_stream_client.rs | 34 +-
.../sdk/src/client_wrappers/binary_topic_client.rs | 29 +-
core/sdk/src/client_wrappers/binary_user_client.rs | 35 +-
core/sdk/src/clients/binary_streams.rs | 13 +-
core/sdk/src/clients/binary_topics.rs | 3 +-
core/sdk/src/clients/binary_users.rs | 9 +-
core/sdk/src/clients/client.rs | 7 +-
core/sdk/src/clients/producer.rs | 10 +-
core/sdk/src/driver.rs | 34 +-
core/sdk/src/http/messages.rs | 9 +-
core/sdk/src/http/scripted_server.rs | 7 +-
core/sdk/src/http/streams.rs | 22 +-
core/sdk/src/http/topics.rs | 29 +-
core/sdk/src/http/users.rs | 19 +-
core/sdk/src/links/fake_cluster.rs | 7 +-
core/sdk/src/prelude.rs | 22 +-
core/sdk/src/session.rs | 19 +
.../src/stream_builder/build/build_stream_topic.rs | 7 +-
core/sdk/src/tcp/tcp_client.rs | 59 +-
core/sdk/src/vsr.rs | 94 +-
core/server/Cargo.toml | 2 +-
core/server/README.md | 67 +-
core/server/config.toml | 32 +-
core/server/src/boot/mod.rs | 45 +-
core/server/src/boot/recovery.rs | 452 ++++-
core/server/src/boot/threads.rs | 2 +
core/server/src/consumer_group.rs | 2 +-
core/server/src/dispatch/authz.rs | 8 +-
core/server/src/dispatch/mod.rs | 172 +-
core/server/src/dispatch/partition.rs | 121 +-
core/server/src/dispatch/session_ops.rs | 90 +-
core/server/src/dispatch/test_support.rs | 92 +-
core/server/src/http/error.rs | 30 +
core/server/src/http/extractor.rs | 169 +-
core/server/src/http/forward.rs | 6 +-
core/server/src/http/handlers.rs | 360 ++--
core/server/src/http/reads.rs | 40 +-
core/server/src/http/submit.rs | 18 +-
core/server/src/lib.rs | 2 +
core/server/src/partition_helpers.rs | 2 +-
core/server/src/partition_reconciler.rs | 462 ++++-
core/server/src/reply_frame.rs | 51 +-
core/server/src/responses.rs | 672 ++++----
core/server/src/rewrite.rs | 1024 +++++++++--
core/server/src/server_error.rs | 19 +
core/server/src/stream_key_rewrapper.rs | 673 ++++++++
core/server/src/stream_keys.rs | 450 +++++
core/server/src/users.rs | 281 ++-
core/server/src/wire.rs | 23 +-
core/shard/src/coordinator.rs | 2 +-
core/shard/src/lib.rs | 11 +-
core/shard/src/partition_io.rs | 2 +-
core/shard/src/poll.rs | 2 +
core/shard/src/poll/completion.rs | 2 +-
core/shard/src/poll/test_support.rs | 2 +-
core/shard/src/primary_rebalance/tests.rs | 2 +-
core/simulator/src/client.rs | 178 +-
core/simulator/src/lib.rs | 238 ++-
core/simulator/src/replica.rs | 10 +-
core/simulator/src/storage/bookmark_recovery.rs | 2 +-
core/simulator/src/workload/actions.rs | 1 +
.../{update_stream.rs => create_keyed_stream.rs} | 50 +-
core/simulator/src/workload/ops/mod.rs | 2 +
core/simulator/src/workload/ops/update_stream.rs | 2 +-
core/simulator/src/workload/ops/update_topic.rs | 50 +-
core/simulator/src/workload/ops/update_user.rs | 58 +-
core/simulator/src/workload/options.rs | 3 +-
core/simulator/src/workload/oracle.rs | 120 +-
core/tools/src/data-seeder/seeder.rs | 12 +-
.../Iggy_SDK.Examples.TcpTls.Producer/Program.cs | 3 +-
examples/go/getting-started/producer/main.go | 9 +-
.../apache/iggy/examples/async/AsyncProducer.java | 10 +-
.../producer/GettingStartedProducer.java | 5 +-
.../producer/MessageEnvelopeProducer.java | 5 +-
.../producer/MessageHeadersProducer.java | 6 +-
.../multitenant/producer/MultiTenantProducer.java | 11 +-
.../sinkdataproducer/SinkDataProducer.java | 10 +-
.../iggy/examples/streambuilder/StreamBasic.java | 5 +-
.../examples/tcptls/producer/TcpTlsProducer.java | 5 +-
.../iggy/examples/MultiTenantAccessTest.java | 26 +-
examples/node/README.md | 2 +-
examples/node/package-lock.json | 7 +-
examples/node/src/stream-builder/example.ts | 2 +-
examples/python/basic/producer.py | 6 +-
examples/python/getting-started/producer.py | 6 +-
examples/python/high-level/producer.py | 10 +-
examples/python/message-headers/common.py | 6 +-
examples/python/uv.lock | 2 +-
examples/rust/src/getting-started/producer/main.rs | 5 +-
.../message-compression/producer/main.rs | 2 +-
examples/rust/src/multi-tenant/consumer/main.rs | 8 +-
examples/rust/src/multi-tenant/producer/main.rs | 12 +-
examples/rust/src/shared/system.rs | 4 +-
examples/rust/src/tcp-tls/producer/main.rs | 5 +-
foreign/cpp/BUILD.bazel | 9 +
foreign/cpp/Cargo.toml | 5 +-
foreign/cpp/MODULE.bazel | 3 +-
foreign/cpp/README.md | 25 +
foreign/cpp/include/iggy.hpp | 948 +++++++++--
foreign/cpp/src/client.cpp | 144 +-
foreign/cpp/src/client.rs | 179 +-
foreign/cpp/src/lib.rs | 119 +-
foreign/cpp/src/type_conversion.rs | 387 ++++-
foreign/cpp/src/type_conversions.cpp | 48 +-
foreign/cpp/tests/e2e/client.cpp | 171 +-
foreign/cpp/tests/e2e/message.cpp | 76 +-
foreign/cpp/tests/e2e/resource_options.cpp | 736 ++++++++
foreign/cpp/tests/e2e/stream.cpp | 38 +-
foreign/cpp/tests/e2e/test_helpers.hpp | 17 +-
foreign/cpp/tests/e2e/topic.cpp | 206 ++-
foreign/cpp/tests/fixtures/resource_options.json | 1619 ++++++++++++++++++
foreign/cpp/tests/unit/resource_options.cpp | 355 ++++
foreign/cpp/tests/unit/unit_tests.cpp | 91 +-
foreign/csharp/Benchmarks/Program.cs | 7 +-
.../Iggy_SDK.Tests.BDD/Iggy_SDK.Tests.BDD.csproj | 10 +
.../StepDefinitions/LeaderRedirectionSteps.cs | 12 +
.../StepDefinitions/ResourceOptionsSteps.cs | 376 ++++
.../csharp/Iggy_SDK.Tests.BDD/xunit.runner.json | 4 +
.../Fixtures/IggyServerFixture.cs | 10 +-
.../Iggy_SDK.Tests.Integration/OptionsTests.cs | 313 +++-
.../Iggy_SDK.Tests.Integration/StreamsTests.cs | 6 +-
.../Iggy_SDK.Tests.Integration/TopicsTests.cs | 34 +-
.../Iggy_SDK.Tests.Integration/UsersTests.cs | 3 +-
.../csharp/Iggy_SDK/Contracts/Auth/UserResponse.cs | 25 +-
.../Contracts/Http/Auth/CreateUserRequest.cs | 9 +-
.../Iggy_SDK/Contracts/Http/CreateStreamRequest.cs | 11 +-
.../Iggy_SDK/Contracts/Http/CreateTopicRequest.cs | 47 +-
.../csharp/Iggy_SDK/Contracts/Http/OptionInputs.cs | 53 +
.../Iggy_SDK/Contracts/Http/UpdateStreamRequest.cs | 20 -
foreign/csharp/Iggy_SDK/Contracts/OptionKeys.cs | 241 +++
foreign/csharp/Iggy_SDK/Contracts/OptionSpec.cs | 38 +-
foreign/csharp/Iggy_SDK/Contracts/OptionValues.cs | 62 +
foreign/csharp/Iggy_SDK/Contracts/OptionsPatch.cs | 53 +
.../UpdateTopicRequest.cs => StreamOptionKey.cs} | 31 +-
foreign/csharp/Iggy_SDK/Contracts/StreamOptions.cs | 74 +
.../csharp/Iggy_SDK/Contracts/StreamResponse.cs | 56 +-
foreign/csharp/Iggy_SDK/Contracts/StreamUpdate.cs | 64 +
.../csharp/Iggy_SDK/Contracts/Tcp/TcpContracts.cs | 221 +--
.../csharp/Iggy_SDK/Contracts/TopicOptionKey.cs | 58 +
foreign/csharp/Iggy_SDK/Contracts/TopicOptions.cs | 139 +-
foreign/csharp/Iggy_SDK/Contracts/TopicResponse.cs | 34 +-
foreign/csharp/Iggy_SDK/Contracts/TopicUpdate.cs | 88 +
.../Auth/CreateUserRequest.cs => UserOptionKey.cs} | 16 +-
.../{Auth/UserResponse.cs => UserOptions.cs} | 48 +-
foreign/csharp/Iggy_SDK/Contracts/UserUpdate.cs | 78 +
.../csharp/Iggy_SDK/Encryption/EncryptionKeys.cs | 108 ++
.../PartitionResizePolicy.cs} | 12 +-
.../StreamEncryptionSource.cs} | 16 +-
foreign/csharp/Iggy_SDK/IggyClient/IIggyStream.cs | 17 +-
foreign/csharp/Iggy_SDK/IggyClient/IIggySystem.cs | 12 +-
foreign/csharp/Iggy_SDK/IggyClient/IIggyTopic.cs | 41 +-
foreign/csharp/Iggy_SDK/IggyClient/IIggyUsers.cs | 16 +-
.../Implementations/HttpMessageStream.cs | 111 +-
.../TcpMessageStream.PollRouting.cs | 48 +-
.../IggyClient/Implementations/TcpMessageStream.cs | 47 +-
foreign/csharp/Iggy_SDK/Iggy_SDK.csproj | 2 +-
.../JsonConverters/RedactedOptionsConverter.cs | 49 +
.../JsonConverters/ResourceOptionsConverter.cs | 133 +-
foreign/csharp/Iggy_SDK/Mappers/BinaryMapper.cs | 57 +-
.../csharp/Iggy_SDK/Publishers/IggyPublisher.cs | 24 +-
foreign/csharp/Iggy_SDK/Utils/CommandCodes.cs | 7 +-
foreign/csharp/Iggy_SDK/Vsr/ConsensusSession.cs | 23 +-
foreign/csharp/Iggy_SDK/Vsr/LoginRegister.cs | 4 +-
foreign/csharp/Iggy_SDK/Vsr/VsrConnection.cs | 7 +-
foreign/csharp/Iggy_SDK/Vsr/VsrError.cs | 8 +
foreign/csharp/Iggy_SDK/Vsr/VsrHeader.cs | 7 +-
foreign/csharp/Iggy_SDK/Vsr/VsrOperation.cs | 51 +-
.../ClientTests/ExampleHelpersTests.cs | 5 +-
.../ClientTests/HttpPartitionContextTests.cs | 2 +-
.../ClientTests/HttpTopicOptionsTests.cs | 28 +-
.../ConsumerTests/PartitionIncarnationTests.cs | 6 +-
.../ContractsTests/TopicContractsTests.cs | 29 +-
.../ContractsTests/WireNameLengthContractsTests.cs | 62 +-
.../Iggy_SDK_Tests/Fixtures/resource_options.json | 1619 ++++++++++++++++++
.../csharp/Iggy_SDK_Tests/Iggy_SDK_Tests.csproj | 6 +
.../Iggy_SDK_Tests/MapperTests/BinaryMapper.cs | 64 +-
.../MapperTests/OptionsBlockGoldenVectorTests.cs | 15 +-
.../MapperTests/ResourceOptionsConverterTests.cs | 37 +-
.../MapperTests/ResourceOptionsTests.cs | 1187 +++++++++++++
.../PublisherTests/IggyTypedPublisherTests.cs | 1 -
.../csharp/Iggy_SDK_Tests/Utils/BinaryFactory.cs | 32 +-
.../Iggy_SDK_Tests/Utils/Users/UsersFactory.cs | 3 +-
.../VsrTests/ConsensusSessionTests.cs | 27 +-
.../VsrTests/EndpointFailoverTests.cs | 3 +-
.../VsrTests/NativeRecoveryTests.SignIn.cs | 2 +-
.../Iggy_SDK_Tests/VsrTests/NativeRecoveryTests.cs | 70 +-
.../VsrTests/PartitionContextTests.cs | 7 +-
.../Iggy_SDK_Tests/VsrTests/PrimaryPollingTests.cs | 7 +-
.../VsrTests/VsrMultiplexingTests.cs | 72 +-
.../Iggy_SDK_Tests/VsrTests/VsrOperationTests.cs | 18 +-
.../VsrTests/VsrProtocolDriftTests.cs | 45 +
foreign/csharp/README.md | 43 +-
foreign/go/README.md | 53 +-
.../go/benchmarks/send_messages_benchmark_test.go | 16 +-
.../binary_response_deserializer.go | 147 +-
.../binary_response_deserializer_test.go | 78 +-
.../vsr_response_deserializer_test.go | 68 +-
foreign/go/client/example_resource_options_test.go | 87 +
.../go/client/tcp/tcp_access_token_management.go | 2 +-
foreign/go/client/tcp/tcp_core_test.go | 2 +-
foreign/go/client/tcp/tcp_link.go | 3 +-
foreign/go/client/tcp/tcp_link_test.go | 17 +
foreign/go/client/tcp/tcp_poll_routing_test.go | 4 +-
.../go/client/tcp/tcp_session_credentials_test.go | 112 +-
foreign/go/client/tcp/tcp_stream_management.go | 21 +-
.../go/client/tcp/tcp_stream_management_test.go | 40 +-
foreign/go/client/tcp/tcp_topic_management.go | 46 +-
foreign/go/client/tcp/tcp_topic_management_test.go | 38 +-
foreign/go/client/tcp/tcp_user_management.go | 89 +-
foreign/go/contracts/client.go | 61 +-
foreign/go/contracts/option_diagnostics.go | 150 ++
foreign/go/contracts/options.go | 22 +-
foreign/go/contracts/resource_options_test.go | 988 +++++++++++
foreign/go/contracts/stream.go | 53 +-
foreign/go/contracts/stream_options.go | 138 ++
.../go/contracts/testdata/resource_options.json | 1619 ++++++++++++++++++
foreign/go/contracts/topic_options.go | 130 +-
foreign/go/contracts/topics.go | 12 +-
foreign/go/contracts/user_options.go | 59 +
foreign/go/contracts/users.go | 13 +-
foreign/go/contracts/version.go | 2 +-
foreign/go/errors/errors.yaml | 32 +
foreign/go/errors/errors_gen.go | 108 ++
foreign/go/internal/command/access_token.go | 12 +-
foreign/go/internal/command/access_token_test.go | 50 +
foreign/go/internal/command/code.go | 6 +-
foreign/go/internal/command/stream.go | 89 +-
foreign/go/internal/command/stream_test.go | 16 +-
foreign/go/internal/command/topic.go | 228 +--
foreign/go/internal/command/topic_test.go | 100 +-
foreign/go/internal/command/update_resource.go | 318 ++++
.../go/internal/command/update_resource_test.go | 181 ++
foreign/go/internal/command/update_user.go | 94 +-
foreign/go/internal/command/update_user_test.go | 43 +-
foreign/go/internal/command/user.go | 9 +-
foreign/go/internal/vsr/envelope.go | 12 +-
foreign/go/internal/vsr/envelope_test.go | 12 +-
foreign/go/internal/vsr/operation.go | 23 +-
foreign/go/internal/vsr/operation_test.go | 17 +-
foreign/go/internal/vsr/protocol_parity_test.go | 8 +-
foreign/go/internal/vsr/register.go | 4 +-
foreign/go/internal/vsr/register_test.go | 6 +-
foreign/go/internal/vsr/reply.go | 4 +-
foreign/go/internal/vsr/reply_test.go | 41 +-
foreign/go/tests/e2e_helpers_test.go | 9 +-
foreign/go/tests/e2e_personal_access_token_test.go | 98 ++
foreign/go/tests/e2e_resource_options_test.go | 301 ++++
foreign/go/tests/e2e_test.go | 59 +-
foreign/java/README.md | 46 +-
.../common/provision/ResourceProvisioner.java | 11 +-
.../connector/pinot/IggyPinotIntegrationTest.java | 6 +-
foreign/java/gradle.properties | 2 +-
foreign/java/java-sdk/build.gradle.kts | 2 +-
.../apache/iggy/client/async/StreamsClient.java | 52 +-
.../org/apache/iggy/client/async/TopicsClient.java | 116 +-
.../org/apache/iggy/client/async/UsersClient.java | 37 +-
.../apache/iggy/client/async/tcp/SessionLogin.java | 14 +-
.../iggy/client/async/tcp/StreamsTcpClient.java | 17 +-
.../iggy/client/async/tcp/TopicsTcpClient.java | 99 +-
.../client/async/tcp/UserPatchCredentials.java | 88 +
.../iggy/client/async/tcp/UsersTcpClient.java | 57 +-
.../client/async/tcp/vsr/ConsensusSession.java | 14 +
.../iggy/client/async/tcp/vsr/VsrLoginCodec.java | 2 +-
.../iggy/client/async/tcp/vsr/VsrOperation.java | 28 +-
.../client/async/tcp/vsr/VsrRequestEncoder.java | 19 +
.../apache/iggy/client/blocking/StreamsClient.java | 32 +-
.../apache/iggy/client/blocking/TopicsClient.java | 87 +-
.../apache/iggy/client/blocking/UsersClient.java | 21 +-
.../iggy/client/blocking/http/HttpOptionValue.java | 164 ++
.../client/blocking/http/InternalHttpClient.java | 2 +-
.../client/blocking/http/StreamsHttpClient.java | 60 +-
.../client/blocking/http/SystemHttpClient.java | 31 +-
.../client/blocking/http/TopicsHttpClient.java | 99 +-
.../iggy/client/blocking/http/UsersHttpClient.java | 77 +-
.../iggy/client/blocking/tcp/StreamsTcpClient.java | 11 +-
.../iggy/client/blocking/tcp/TopicsTcpClient.java | 26 +-
.../iggy/client/blocking/tcp/UsersTcpClient.java | 14 +-
.../org/apache/iggy/exception/IggyErrorCode.java | 14 +
.../iggy/exception/IggyValidationException.java | 9 +-
.../org/apache/iggy/serde/BytesDeserializer.java | 45 +-
.../org/apache/iggy/serde/BytesSerializer.java | 144 ++
.../java/org/apache/iggy/serde/CommandCode.java | 11 +-
.../apache/iggy/stream/EncryptionKeySource.java | 52 +
.../org/apache/iggy/stream/EncryptionKeys.java | 120 ++
.../apache/iggy/stream/EncryptionProjection.java | 60 +
.../java/org/apache/iggy/stream/StreamBase.java | 31 +-
.../java/org/apache/iggy/stream/StreamDetails.java | 42 +-
.../org/apache/iggy/stream/StreamOptionKey.java | 59 +
.../java/org/apache/iggy/stream/StreamOptions.java | 128 ++
.../java/org/apache/iggy/stream/StreamUpdate.java | 138 ++
.../java/org/apache/iggy/system/OptionKey.java | 177 ++
.../java/org/apache/iggy/system/OptionSpec.java | 47 +-
.../PartitionResizePolicy.java} | 18 +-
.../src/main/java/org/apache/iggy/topic/Topic.java | 33 +-
.../java/org/apache/iggy/topic/TopicDetails.java | 22 +-
.../java/org/apache/iggy/topic/TopicOptionKey.java | 63 +
.../java/org/apache/iggy/topic/TopicOptions.java | 123 +-
.../java/org/apache/iggy/topic/TopicSettings.java | 67 +
.../java/org/apache/iggy/topic/TopicUpdate.java | 174 ++
.../main/java/org/apache/iggy/user/UserInfo.java | 22 +-
.../java/org/apache/iggy/user/UserInfoDetails.java | 26 +-
.../{UserInfoDetails.java => UserOptionKey.java} | 32 +-
.../java/org/apache/iggy/user/UserOptions.java | 74 +
.../main/java/org/apache/iggy/user/UserUpdate.java | 163 ++
.../java/org/apache/iggy/ProtocolParityTest.java | 4 +-
.../org/apache/iggy/ResourceOptionsFixture.java | 201 +++
.../test/java/org/apache/iggy/TestConstants.java | 5 +-
.../client/async/AsyncClientIntegrationTest.java | 37 +-
.../client/async/AsyncConnectionPoolAuthTest.java | 20 +-
.../iggy/client/async/AsyncConsumerGroupsTest.java | 10 +-
.../iggy/client/async/ResourceOptionsTest.java | 371 ++++
.../AsyncIggyTcpClientCredentialChangeTest.java | 293 ++++
.../AsyncIggyTcpClientEndpointFailoverTest.java | 185 +-
.../tcp/AsyncIggyTcpClientEventLoopGroupTest.java | 2 +-
.../AsyncIggyTcpClientTransientFailoverTest.java | 83 +-
.../tcp/AsyncTcpConnectionConcurrencyTest.java | 2 +-
.../async/tcp/TopicsTcpClientPayloadTest.java | 42 +-
.../client/async/tcp/vsr/VsrOperationTest.java | 69 +
.../async/tcp/vsr/VsrRequestEncoderTest.java | 14 +
.../async/tcp/vsr/VsrResponseHandlerTest.java | 53 +
.../iggy/client/blocking/IntegrationTest.java | 12 +-
.../iggy/client/blocking/StreamClientBaseTest.java | 155 +-
.../iggy/client/blocking/SystemClientBaseTest.java | 43 +-
.../iggy/client/blocking/TopicsClientBaseTest.java | 126 +-
.../iggy/client/blocking/UsersClientBaseTest.java | 66 +-
.../client/blocking/http/HttpOptionValueTest.java | 215 +++
.../http/InternalHttpClientLoggingTest.java | 61 +
.../http/MessagesHttpClientSendContextTest.java | 4 +-
.../client/blocking/http/StreamHttpClientTest.java | 37 +
.../client/blocking/tcp/MessagesTcpClientTest.java | 12 +-
.../client/blocking/tcp/TlsConnectionTest.java | 4 +-
.../client/blocking/tcp/TopicsTcpClientTest.java | 18 +-
.../exception/IggyValidationExceptionTest.java | 18 +-
.../apache/iggy/serde/BytesDeserializerTest.java | 16 +-
.../org/apache/iggy/serde/ResourceOptionsTest.java | 670 ++++++++
.../org/apache/iggy/stream/StreamDetailsTest.java | 21 +-
.../org/apache/iggy/topic/TopicDetailsTest.java | 18 +-
.../org/apache/iggy/topic/TopicOptionsTest.java | 15 +-
.../org/apache/iggy/user/UserInfoDetailsTest.java | 2 +-
.../src/test/resources/resource_options.json | 1619 ++++++++++++++++++
foreign/node/README.md | 49 +-
foreign/node/package-lock.json | 4 +-
foreign/node/package.json | 2 +-
foreign/node/scripts/check-vsr-protocol.mjs | 22 +
foreign/node/src/bdd/README.md | 2 +
foreign/node/src/bdd/resource-options.ts | 333 ++++
foreign/node/src/bdd/world.ts | 7 +-
foreign/node/src/client/client.operation.test.ts | 15 +
foreign/node/src/client/client.socket.test.ts | 102 +-
foreign/node/src/client/client.socket.ts | 22 +-
.../src/e2e/resource-options.e2e.ts} | 24 +-
foreign/node/src/e2e/resource-options.suite.ts | 323 ++++
foreign/node/src/e2e/tcp.token.e2e.ts | 10 +-
foreign/node/src/e2e/tls.system.e2e.ts | 20 +-
foreign/node/src/index.ts | 17 +
foreign/node/src/wire/command.code.ts | 7 +-
foreign/node/src/wire/error.code.test.ts | 9 +
foreign/node/src/wire/error.code.ts | 6 +
.../node/src/wire/fixtures/resource_options.json | 1619 ++++++++++++++++++
foreign/node/src/wire/index.ts | 4 +
foreign/node/src/wire/options.utils.ts | 80 +
.../node/src/wire/stream/create-stream.command.ts | 33 +-
foreign/node/src/wire/stream/index.ts | 1 +
.../node/src/wire/stream/stream-options.test.ts | 113 ++
foreign/node/src/wire/stream/stream-options.ts | 140 ++
foreign/node/src/wire/stream/stream.utils.ts | 31 +-
.../node/src/wire/stream/update-stream.command.ts | 52 +-
.../src/wire/system/describe-options.command.ts | 43 +-
foreign/node/src/wire/system/index.ts | 1 +
.../wire/system/update-resource.command.test.ts | 680 ++++++++
.../src/wire/system/update-resource.command.ts | 173 ++
.../src/wire/token/create-token.command.test.ts | 14 +
.../node/src/wire/token/create-token.command.ts | 4 +-
.../node/src/wire/topic/create-topic.command.ts | 43 +-
foreign/node/src/wire/topic/index.ts | 2 +-
foreign/node/src/wire/topic/topic.utils.test.ts | 26 +-
foreign/node/src/wire/topic/topic.utils.ts | 44 +-
.../node/src/wire/topic/update-topic.command.ts | 87 +-
foreign/node/src/wire/user/create-user.command.ts | 23 +-
foreign/node/src/wire/user/index.ts | 1 +
foreign/node/src/wire/user/update-user.command.ts | 82 +-
foreign/node/src/wire/user/user.utils.ts | 13 +-
foreign/node/src/wire/vsr/index.ts | 12 +-
foreign/node/src/wire/vsr/operation.test.ts | 33 +-
foreign/node/src/wire/vsr/operation.ts | 41 +-
foreign/node/src/wire/vsr/register.ts | 4 +-
foreign/node/src/wire/vsr/reply.test.ts | 16 +-
foreign/node/src/wire/vsr/reply.ts | 9 +-
foreign/node/src/wire/vsr/session.ts | 11 +
foreign/node/src/wire/vsr/vsr.test.ts | 21 +-
foreign/php/Cargo.toml | 3 +-
foreign/php/README.md | 42 +-
foreign/php/docker-compose.test.yml | 4 +
foreign/php/iggy-php.stubs.php | 246 ++-
foreign/php/src/client.rs | 202 ++-
foreign/php/src/error.rs | 22 +-
foreign/php/src/lib.rs | 8 +
foreign/php/src/options.rs | 338 ++++
foreign/php/src/sensitive_parameters.rs | 135 ++
foreign/php/src/stream.rs | 44 +-
foreign/php/src/topic.rs | 18 +-
foreign/php/tests/ErrorTest.php | 18 +
foreign/php/tests/ResourceOptionsTest.php | 640 +++++++
foreign/php/tests/SensitiveParameterTest.php | 166 ++
foreign/php/tests/bootstrap.php | 28 +
foreign/php/tests/fixtures/resource_options.json | 1619 ++++++++++++++++++
foreign/python/Cargo.toml | 10 +-
foreign/python/README.md | 61 +-
foreign/python/apache_iggy.pyi | 556 +++++-
foreign/python/docker-compose.test.yml | 3 +
foreign/python/pyproject.toml | 4 +-
foreign/python/src/client.rs | 386 +++--
foreign/python/src/lib.rs | 14 +-
foreign/python/src/options.rs | 64 +-
foreign/python/src/resource_options.rs | 614 +++++++
foreign/python/src/stream.rs | 90 +
foreign/python/src/topic.rs | 67 +-
foreign/python/src/user.rs | 36 +
foreign/python/tests/conftest.py | 35 +-
.../python/tests/fixtures/resource_options.json | 1619 ++++++++++++++++++
foreign/python/tests/test_client_config.py | 6 +-
foreign/python/tests/test_consumer_group.py | 267 +--
foreign/python/tests/test_http_config.py | 8 +-
foreign/python/tests/test_message_operations.py | 131 +-
foreign/python/tests/test_partition.py | 5 +-
foreign/python/tests/test_permissions.py | 2 +-
foreign/python/tests/test_producer.py | 11 +-
foreign/python/tests/test_resource_options.py | 742 ++++++++
foreign/python/tests/test_segment.py | 45 +-
foreign/python/tests/test_stats.py | 14 +-
foreign/python/tests/test_stream.py | 4 +-
foreign/python/tests/test_tls.py | 2 +-
foreign/python/tests/test_topic.py | 290 ++--
foreign/python/tests/test_user.py | 9 +-
foreign/python/uv.lock | 11 +-
foreign/swift/Package.resolved | 20 +-
foreign/swift/Package.swift | 2 +
foreign/swift/README.md | 45 +-
foreign/swift/Sources/Iggy/Client/ClientCore.swift | 7 +-
.../Sources/Iggy/Client/IggyClient+Resources.swift | 41 +-
foreign/swift/Sources/Iggy/Client/IggyClient.swift | 4 +-
.../swift/Sources/Iggy/Errors/IggyErrorCode.swift | 12 +
foreign/swift/Sources/Iggy/Iggy.swift | 2 +-
.../swift/Sources/Iggy/Models/EncryptionKey.swift | 65 +
.../Sources/Iggy/Models/ResourceOptions.swift | 116 +-
foreign/swift/Sources/Iggy/Models/Resources.swift | 22 +-
.../swift/Sources/Iggy/Models/StreamOptions.swift | 191 +++
.../swift/Sources/Iggy/Models/TopicOptions.swift | 195 ++-
.../swift/Sources/Iggy/Models/UserHeaders.swift | 2 +-
.../swift/Sources/Iggy/Models/UserOptions.swift | 88 +
.../swift/Sources/Iggy/Protocol/CommandCode.swift | 13 +-
.../Sources/Iggy/Protocol/ConsensusOperation.swift | 27 +-
.../Sources/Iggy/Protocol/ConsensusSession.swift | 2 +
.../Sources/Iggy/Protocol/ProtocolVersion.swift | 2 +-
foreign/swift/Sources/Iggy/Protocol/Requests.swift | 78 +-
.../swift/Sources/Iggy/Protocol/Responses.swift | 26 +-
foreign/swift/Sources/Iggy/Protocol/VSRFrame.swift | 20 +-
.../swift/Tests/IggyE2ETests/ClientE2ETests.swift | 41 +-
.../IggyE2ETests/ResourceOptionsE2ETests.swift | 309 ++++
.../IggyE2ETests/Support/E2EEnvironment.swift | 3 +
.../Tests/IggyTests/ClientTransportTests.swift | 25 +-
foreign/swift/Tests/IggyTests/Fixtures/golden.json | 70 +-
.../Tests/IggyTests/Fixtures/resource_options.json | 1619 ++++++++++++++++++
foreign/swift/Tests/IggyTests/ModelTests.swift | 30 +-
.../swift/Tests/IggyTests/ModelVectorTests.swift | 29 +-
foreign/swift/Tests/IggyTests/ProtocolTests.swift | 26 +-
.../Tests/IggyTests/ResourceOptionsTests.swift | 586 +++++++
.../Tests/IggyTests/ResponseVectorTests.swift | 22 +-
.../Tests/IggyTests/Support/FakeVSRServer.swift | 8 +
.../IggyTests/Support/ResourceOptionsFixture.swift | 412 +++++
.../Tests/IggyTests/Support/TestHelpers.swift | 10 +-
.../swift/Tests/IggyTests/WireVectorTests.swift | 61 +-
foreign/swift/Tools/golden-vectors/src/main.rs | 237 ++-
.../kafka/src/bridge/iggy_bridge/cluster_id.rs | 12 +-
gateways/kafka/src/bridge/iggy_bridge/topics.rs | 20 +-
.../src/protocol/handlers/describe_configs.rs | 24 +-
.../kafka/tests/bridge_iggy_integration_tests.rs | 3 +-
.../kafka/tests/cluster_id_real_bridge_tests.rs | 17 +-
gateways/kafka/tests/metadata_real_bridge_tests.rs | 10 +-
.../kafka/tests/offset_groups_real_bridge_tests.rs | 3 +-
helm/charts/iggy/Chart.yaml | 2 +-
helm/charts/iggy/README.md | 22 +-
helm/charts/iggy/README.md.gotmpl | 12 +-
helm/charts/iggy/examples/cluster-3-node.yaml | 5 +-
helm/charts/iggy/templates/_helpers.tpl | 49 +-
helm/charts/iggy/templates/server-secrets.yaml | 5 +-
helm/charts/iggy/values.yaml | 20 +-
scripts/ci/test-helm.sh | 77 +
scripts/run-bdd-tests.sh | 11 +-
web/docs/server-api.http | 57 +-
web/package-lock.json | 4 +-
web/package.json | 2 +-
web/src/lib/api/ApiSchema.ts | 40 +-
web/src/lib/api/resourceOptions.test.mjs | 159 ++
web/src/lib/api/resourceOptions.ts | 146 ++
.../lib/components/Modals/AddStreamModal.svelte | 39 +-
web/src/lib/components/Modals/AddTopicModal.svelte | 37 +-
web/src/lib/components/Modals/EditUserModal.svelte | 116 +-
.../components/Modals/StreamSettingsModal.svelte | 30 +-
.../components/Modals/TopicSettingsModal.svelte | 100 +-
.../RouteComponents/Settings/UsersTab.svelte | 8 +-
web/src/lib/domain/Stream.ts | 11 +-
web/src/lib/domain/Topic.ts | 25 +-
.../routes/dashboard/settings/users/+page.svelte | 9 +-
820 files changed, 68010 insertions(+), 9137 deletions(-)
diff --git a/.dockerignore b/.dockerignore
index b4405acc2..30d0a2bb6 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -10,15 +10,21 @@
/helm
/licenses
/local_data
+/local_state
/performance_results
/test_logs
+**/.venv
+**/node_modules
+**/.gradle
/foreign/swift/.build
+/foreign/java/**/build
/bdd/java/build
/examples/java/**/build
-/foreign/java/**/build
/web/build
-**/.gradle
-**/node_modules
+/foreign/csharp/**/bin
+/foreign/csharp/**/obj
+/foreign/cpp/bazel-*
+/foreign/php/vendor
# Selectively exclude scripts/ subdirs not needed in container builds.
# scripts/ci/third-party-licenses.sh + render-node-licenses.mjs are
# required by the convenience-binary Dockerfiles to render LICENSE-binary
diff --git a/.github/actions/cpp-bazel/pre-merge/action.yml b/.github/actions/cpp-bazel/pre-merge/action.yml
index ebfc34830..8913da1c3 100644
--- a/.github/actions/cpp-bazel/pre-merge/action.yml
+++ b/.github/actions/cpp-bazel/pre-merge/action.yml
@@ -88,6 +88,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
- name: Test Rust bridge and C++ coverage
if: inputs.task == 'test'
@@ -104,6 +106,37 @@ runs:
pid-file: ${{ steps.iggy.outputs.pid_file }}
log-file: ${{ steps.iggy.outputs.log_file }}
+ - name: Setup server on the shipped default
+ id: iggy-default
+ if: inputs.task == 'test'
+ uses: ./.github/actions/utils/server-start
+ with:
+ bin: target/debug/iggy-server
+ pid-file: ${{ runner.temp }}/iggy-server-default.pid
+ log-file: ${{ runner.temp }}/iggy-server-default.log
+ env:
+ IGGY_PATH: ${{ runner.temp }}/iggy-cpp-default-data
+
+ # The resource options cases need a server key, so they run on the lane above.
+ - name: Test C++ e2e on the shipped default
+ if: inputs.task == 'test'
+ shell: bash
+ run: |
+ cd foreign/cpp
+ bazel test \
+ --config=debug \
+ --lockfile_mode=error \
+ --strategy=TestRunner=standalone \
+ --test_arg=--gtest_filter=-*ResourceOptions* \
+ //:e2e
+
+ - name: Stop server after shipped default tests
+ if: always() && inputs.task == 'test'
+ uses: ./.github/actions/utils/server-stop
+ with:
+ pid-file: ${{ steps.iggy-default.outputs.pid_file }}
+ log-file: ${{ steps.iggy-default.outputs.log_file }}
+
- name: Validate task
if: inputs.task != 'lint' && inputs.task != 'build' && inputs.task != 'test'
shell: bash
diff --git a/.github/actions/go/pre-merge/action.yml b/.github/actions/go/pre-merge/action.yml
index ebf640680..a04736c83 100644
--- a/.github/actions/go/pre-merge/action.yml
+++ b/.github/actions/go/pre-merge/action.yml
@@ -129,6 +129,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
replica-id: "0"
pid-file: ${{ runner.temp }}/iggy-go-e2e.pid
log-file: ${{ runner.temp }}/iggy-go-e2e.log
@@ -179,12 +181,45 @@ runs:
pid-file: ${{ steps.iggy.outputs.pid_file }}
log-file: ${{ steps.iggy.outputs.log_file }}
+ - name: Start Iggy server (shipped default)
+ id: iggy-default
+ if: inputs.task == 'e2e'
+ uses: ./.github/actions/utils/server-start
+ with:
+ bin: target/debug/iggy-server
+ replica-id: "0"
+ pid-file: ${{ runner.temp }}/iggy-go-e2e-default.pid
+ log-file: ${{ runner.temp }}/iggy-go-e2e-default.log
+ wait-timeout-seconds: "90"
+ env:
+ IGGY_PATH: ${{ runner.temp }}/iggy-go-e2e-default-data
+
+ # The resource options cases need a server key, so they run on the other lanes.
+ - name: Run e2e tests (shipped default)
+ shell: bash
+ if: inputs.task == 'e2e'
+ env:
+ IGGY_TCP_ADDRESS: 127.0.0.1:8090
+ run: |
+ cd foreign/go
+ go test -v -race ./tests/... \
+ -skip '^TestE2E_(StreamKeyProjection|KeylessStream|StreamRename|TopicReset|UserOptions)$'
+
+ - name: Stop Iggy server (shipped default)
+ if: always() && inputs.task == 'e2e'
+ uses: ./.github/actions/utils/server-stop
+ with:
+ pid-file: ${{ steps.iggy-default.outputs.pid_file }}
+ log-file: ${{ steps.iggy-default.outputs.log_file }}
+
- name: Start Iggy server (TLS)
id: iggy-tls
if: inputs.task == 'e2e'
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
replica-id: "0"
pid-file: ${{ runner.temp }}/iggy-go-e2e-tls.pid
log-file: ${{ runner.temp }}/iggy-go-e2e-tls.log
@@ -223,6 +258,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
replica-id: "0"
pid-file: ${{ runner.temp }}/iggy-go-cluster-0.pid
log-file: ${{ runner.temp }}/iggy-go-cluster-0.log
@@ -237,6 +274,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
replica-id: "1"
tcp_address: 127.0.0.1:8091
http_address: 127.0.0.1:3001
diff --git a/.github/actions/java-gradle/pre-merge/action.yml b/.github/actions/java-gradle/pre-merge/action.yml
index 35920d0fc..060c3b489 100644
--- a/.github/actions/java-gradle/pre-merge/action.yml
+++ b/.github/actions/java-gradle/pre-merge/action.yml
@@ -86,6 +86,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
wait-timeout-seconds: "90"
env:
IGGY_PATH: ${{ runner.temp }}/iggy-java-data
@@ -156,6 +158,8 @@ runs:
pid-file: ${{ steps.iggy.outputs.pid_file }}
log-file: ${{ steps.iggy.outputs.log_file }}
+ # No encryption key, the shipped default, so this lane also covers a server
+ # that keeps streams plaintext and refuses client stream keys.
- name: Start Iggy server (TLS)
id: iggy-tls
if: inputs.task == 'test'
diff --git a/.github/actions/node-npm/pre-merge/action.yml b/.github/actions/node-npm/pre-merge/action.yml
index 97f0eb88d..1275ba8d5 100644
--- a/.github/actions/node-npm/pre-merge/action.yml
+++ b/.github/actions/node-npm/pre-merge/action.yml
@@ -95,6 +95,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
wait-timeout-seconds: "90"
- name: E2E tests
@@ -120,12 +122,40 @@ runs:
path: ${{ steps.iggy.outputs.log_file }}
if-no-files-found: ignore
+ - name: Start Iggy server (shipped default)
+ id: iggy-default
+ if: inputs.task == 'e2e'
+ uses: ./.github/actions/utils/server-start
+ with:
+ bin: target/debug/iggy-server
+ pid-file: ${{ runner.temp }}/iggy-server-default.pid
+ log-file: ${{ runner.temp }}/iggy-server-default.log
+ env:
+ IGGY_PATH: ${{ runner.temp }}/iggy-node-default-data
+
+ # The resource options suites need a server key, so they run on the lanes above.
+ - name: E2E tests (shipped default)
+ if: inputs.task == 'e2e'
+ run: |
+ cd foreign/node
+ node --import @swc-node/register/esm-register --test --test-force-exit \
+ $(ls src/e2e/*.e2e.ts | grep -v -e resource-options -e tls.system)
+ shell: bash
+
+ - name: Stop Iggy server (shipped default)
+ if: always() && inputs.task == 'e2e'
+ uses: ./.github/actions/utils/server-stop
+ with:
+ pid-file: ${{ steps.iggy-default.outputs.pid_file }}
+ log-file: ${{ steps.iggy-default.outputs.log_file }}
+
- name: Start Iggy server (TLS)
id: iggy-tls
if: inputs.task == 'e2e'
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
pid-file: ${{ runner.temp }}/iggy-server-tls.pid
log-file: ${{ runner.temp }}/iggy-server-tls.log
env:
diff --git a/.github/actions/php/pre-merge/action.yml b/.github/actions/php/pre-merge/action.yml
index 4f88da356..3b8ba1475 100644
--- a/.github/actions/php/pre-merge/action.yml
+++ b/.github/actions/php/pre-merge/action.yml
@@ -34,6 +34,7 @@ runs:
composer \
libclang-dev \
libhwloc-dev \
+ libphp8.5-embed \
libssl-dev \
php8.5-cli \
php8.5-dev \
@@ -102,12 +103,13 @@ runs:
exit 1
fi
raw_stubs="$(mktemp)"
- cargo php stubs "$extension" -o "$raw_stubs"
+ # cargo-php loads the extension outside PHP, so expose PHP's allocator symbols.
+ LD_PRELOAD=/usr/lib/libphp8.5.so cargo php stubs "$extension" -o "$raw_stubs"
{
printf '<?php\n'
awk 'NF { print "// " $0; next } { print "//" }' ASF_LICENSE.txt
printf '\n'
- tail -n +2 "$raw_stubs"
+ tail -n +2 "$raw_stubs" | sed -E 's/(public readonly \?string \$[A-Za-z_][A-Za-z0-9_]*) = null;/\1;/'
} > foreign/php/iggy-php.stubs.php
rm -f "$raw_stubs"
git diff --exit-code -- foreign/php/iggy-php.stubs.php
@@ -165,6 +167,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
- name: Run PHP SDK tests
if: inputs.task == 'test'
@@ -185,6 +189,8 @@ runs:
pid-file: ${{ steps.iggy.outputs.pid_file }}
log-file: ${{ steps.iggy.outputs.log_file }}
+ # No encryption key, the shipped default, so this lane also covers a server
+ # that keeps streams plaintext and refuses client stream keys.
- name: Start Iggy server (TLS)
if: inputs.task == 'test'
id: iggy-tls
@@ -194,6 +200,7 @@ runs:
pid-file: ${{ runner.temp }}/iggy-server-tls.pid
log-file: ${{ runner.temp }}/iggy-server-tls.log
env:
+ IGGY_PATH: ${{ runner.temp }}/iggy-php-tls-data
IGGY_TCP_TLS_ENABLED: "true"
IGGY_TCP_TLS_CERT_FILE: core/certs/iggy_cert.pem
IGGY_TCP_TLS_KEY_FILE: core/certs/iggy_key.pem
diff --git a/.github/actions/python-maturin/pre-merge/action.yml b/.github/actions/python-maturin/pre-merge/action.yml
index 80ca9089f..f71d29ab1 100644
--- a/.github/actions/python-maturin/pre-merge/action.yml
+++ b/.github/actions/python-maturin/pre-merge/action.yml
@@ -54,6 +54,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
- name: Setup Python
uses: actions/setup-python@v6
@@ -214,6 +216,45 @@ runs:
pid-file: ${{ steps.iggy.outputs.pid_file }}
log-file: ${{ steps.iggy.outputs.log_file }}
+ - name: Start Iggy server (shipped default)
+ if: inputs.task == 'test'
+ id: iggy-default
+ uses: ./.github/actions/utils/server-start
+ with:
+ bin: target/debug/iggy-server
+ pid-file: ${{ runner.temp }}/iggy-server-default.pid
+ log-file: ${{ runner.temp }}/iggy-server-default.log
+ env:
+ IGGY_PATH: ${{ runner.temp }}/iggy-python-default-data
+
+ # The resource options cases need a server key, and the TLS cases start
+ # their own server, so both run on the lane above.
+ - name: Run Python integration tests (shipped default)
+ if: inputs.task == 'test'
+ run: |
+ cd foreign/python
+ tcp_port="${{ steps.iggy-default.outputs.tcp_address }}"
+ http_port="${{ steps.iggy-default.outputs.http_address }}"
+ IGGY_SERVER_HOST=127.0.0.1 \
+ IGGY_SERVER_TCP_PORT="${tcp_port##*:}" \
+ IGGY_SERVER_QUIC_PORT=8080 \
+ IGGY_SERVER_HTTP_PORT="${http_port##*:}" \
+ IGGY_SERVER_WS_PORT=8092 \
+ uv run --no-sync pytest tests/ -v \
+ --ignore=tests/test_resource_options.py \
+ --ignore=tests/test_tls.py \
+ --junitxml=../../reports/python-junit-default.xml \
+ --tb=short \
+ --capture=no
+ shell: bash
+
+ - name: Stop Iggy server (shipped default)
+ if: always() && inputs.task == 'test'
+ uses: ./.github/actions/utils/server-stop
+ with:
+ pid-file: ${{ steps.iggy-default.outputs.pid_file }}
+ log-file: ${{ steps.iggy-default.outputs.log_file }}
+
- name: Generate coverage report
if: inputs.task == 'test'
run: |
diff --git a/.github/actions/swift/pre-merge/action.yml b/.github/actions/swift/pre-merge/action.yml
index 077f49ef3..d06030ff3 100644
--- a/.github/actions/swift/pre-merge/action.yml
+++ b/.github/actions/swift/pre-merge/action.yml
@@ -124,6 +124,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
replica-id: "0"
pid-file: ${{ runner.temp }}/iggy-swift-e2e.pid
log-file: ${{ runner.temp }}/iggy-swift-e2e.log
@@ -150,12 +152,46 @@ runs:
pid-file: ${{ steps.iggy.outputs.pid_file }}
log-file: ${{ steps.iggy.outputs.log_file }}
+ - name: Start Iggy server (shipped default)
+ id: iggy-default
+ if: inputs.task == 'e2e'
+ uses: ./.github/actions/utils/server-start
+ with:
+ bin: target/debug/iggy-server
+ replica-id: "0"
+ pid-file: ${{ runner.temp }}/iggy-swift-e2e-default.pid
+ log-file: ${{ runner.temp }}/iggy-swift-e2e-default.log
+ wait-timeout-seconds: "90"
+ env:
+ IGGY_PATH: ${{ runner.temp }}/iggy-swift-e2e-default-data
+
+ # The resource options cases need a server key, so they run on the other lanes.
+ - name: Run e2e tests (shipped default)
+ shell: bash
+ if: inputs.task == 'e2e'
+ env:
+ IGGY_TCP_ADDRESS: 127.0.0.1:8090
+ IGGY_ROOT_USERNAME: iggy
+ IGGY_ROOT_PASSWORD: iggy
+ run: |
+ cd foreign/swift
+ swift test --skip-build --filter IggyE2ETests --skip ResourceOptionsE2ETests
+
+ - name: Stop Iggy server (shipped default)
+ if: always() && inputs.task == 'e2e'
+ uses: ./.github/actions/utils/server-stop
+ with:
+ pid-file: ${{ steps.iggy-default.outputs.pid_file }}
+ log-file: ${{ steps.iggy-default.outputs.log_file }}
+
- name: Start Iggy server (TLS)
id: iggy-tls
if: inputs.task == 'e2e'
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
replica-id: "0"
pid-file: ${{ runner.temp }}/iggy-swift-e2e-tls.pid
log-file: ${{ runner.temp }}/iggy-swift-e2e-tls.log
@@ -197,6 +233,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
replica-id: "0"
pid-file: ${{ runner.temp }}/iggy-swift-cluster-0.pid
log-file: ${{ runner.temp }}/iggy-swift-cluster-0.log
@@ -211,6 +249,8 @@ runs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
replica-id: "1"
tcp_address: 127.0.0.1:8091
http_address: 127.0.0.1:3001
diff --git a/.github/actions/utils/server-start/action.yml b/.github/actions/utils/server-start/action.yml
index 31e6bfe1f..9638ed1b5 100644
--- a/.github/actions/utils/server-start/action.yml
+++ b/.github/actions/utils/server-start/action.yml
@@ -53,6 +53,14 @@ inputs:
description: "--replica-id <u8> value (leave empty to omit)"
required: false
default: ""
+ encryption-enabled:
+ description: "Give streams created without their own key a generated key, which the SDK server-generated key tests need"
+ required: false
+ default: "false"
+ encryption-key:
+ description: "Server encryption key (base64 of 32 bytes). Empty, the shipped default, starts without a key, so streams stay plaintext and client stream keys are refused. Lanes that test stream keys pass the fixed key shared with the BDD server."
+ required: false
+ default: ""
outputs:
pid:
description: "PID of the background server process"
@@ -116,6 +124,13 @@ runs:
export IGGY_HTTP_ADDRESS="${{ inputs.http_address }}"
export IGGY_ROOT_USERNAME="iggy"
export IGGY_ROOT_PASSWORD="iggy"
+ if [[ -n "${{ inputs.encryption-key }}" ]]; then
+ export IGGY_ENCRYPTION_KEY="${{ inputs.encryption-key }}"
+ fi
+ if [[ "${{ inputs.encryption-enabled }}" == "true" ]]; then
+ [[ -n "${{ inputs.encryption-key }}" ]] || { echo "::error::encryption-enabled requires encryption-key"; exit 1; }
+ export IGGY_ENCRYPTION_ENABLED="true"
+ fi
CLI_ARGS=()
if [[ -n "${{ inputs.replica-id }}" ]]; then
CLI_ARGS+=(--replica-id "${{ inputs.replica-id }}")
@@ -124,7 +139,7 @@ runs:
nohup "${{ inputs.bin }}" "${CLI_ARGS[@]}" >"$LOG_FILE" 2>&1 &
echo $! > "$PID_FILE"
echo "Started server PID $(cat "$PID_FILE") → logs: $LOG_FILE"
- echo "Server configured with TCP=$IGGY_TCP_ADDRESS, HTTP=$IGGY_HTTP_ADDRESS"
+ echo "Server configured with TCP=$IGGY_TCP_ADDRESS, HTTP=$IGGY_HTTP_ADDRESS, encryption=${IGGY_ENCRYPTION_ENABLED:-false}"
- name: Wait for readiness
shell: bash
diff --git a/.github/workflows/coverage-baseline.yml b/.github/workflows/coverage-baseline.yml
index aada7fa94..47fa67e04 100644
--- a/.github/workflows/coverage-baseline.yml
+++ b/.github/workflows/coverage-baseline.yml
@@ -148,6 +148,8 @@ jobs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
- name: Run tests and generate coverage
working-directory: foreign/java
@@ -298,6 +300,8 @@ jobs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
- name: Setup Python
uses: actions/setup-python@v7.0.0
@@ -447,6 +451,8 @@ jobs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
wait-timeout-seconds: "90"
- name: Run unit tests with coverage
@@ -499,6 +505,8 @@ jobs:
uses: ./.github/actions/utils/server-start
with:
bin: target/debug/iggy-server
+ encryption-enabled: "true"
+ encryption-key: "/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE="
replica-id: "0"
wait-timeout-seconds: "90"
diff --git a/.gitignore b/.gitignore
index 2b6bf9c31..2dfad0276 100644
--- a/.gitignore
+++ b/.gitignore
@@ -48,6 +48,8 @@ foreign/node/.npm
*.a
*.lib
*.out
+*.pid
+__pycache__/
go.work
core/bench/dashboard/frontend/dist
core/integration/tests/connectors/postgres/cdc_restart_connectors/*
diff --git a/.typos.toml b/.typos.toml
index 64448b23d..420315787 100644
--- a/.typos.toml
+++ b/.typos.toml
@@ -162,5 +162,7 @@ extend-exclude = [
"**/wiremock/__files/*.json",
# Swift SDK golden vectors dumped from the Rust crates (hex payloads)
"foreign/swift/Tests/IggyTests/Fixtures/golden.json",
+ # Resource options fixture copies in every SDK test tree (hex payloads)
+ "**/resource_options.json",
"**/.python-version",
]
diff --git a/Cargo.lock b/Cargo.lock
index ff5a829f9..b2155b6a6 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -249,6 +249,15 @@ dependencies = [
"inout 0.2.2",
]
+[[package]]
+name = "aead-stream"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a901e5bcd15b4a9555a8f17a608d28ef92cf77bc4aa3b4a0c1a3fce1a9cc0bac"
+dependencies = [
+ "aead 0.6.1",
+]
+
[[package]]
name = "aes"
version = "0.8.4"
@@ -269,6 +278,7 @@ dependencies = [
"cipher 0.5.2",
"cpubits",
"cpufeatures 0.3.1",
+ "zeroize",
]
[[package]]
@@ -297,6 +307,22 @@ dependencies = [
"ctr 0.10.1",
"ctutils",
"ghash 0.6.0",
+ "zeroize",
+]
+
+[[package]]
+name = "aes-gcm-siv"
+version = "0.12.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4c32325128cb24de2a2d14d92da7569dae7073f9e47334521d494747fa94d902"
+dependencies = [
+ "aead 0.6.1",
+ "aes 0.9.3",
+ "cipher 0.5.2",
+ "ctr 0.10.1",
+ "ctutils",
+ "polyval 0.7.3",
+ "zeroize",
]
[[package]]
@@ -1431,6 +1457,10 @@ dependencies = [
"futures-util",
"log",
"pin-project-lite",
+ "rustls-native-certs",
+ "rustls-pki-types",
+ "tokio",
+ "tokio-rustls",
"tungstenite 0.29.0",
]
@@ -3203,7 +3233,7 @@ dependencies = [
"compio-log",
"compio-send-wrapper",
"crossbeam-queue",
- "flume 0.12.0",
+ "flume",
"futures-util",
"io-uring",
"libc",
@@ -3324,7 +3354,7 @@ dependencies = [
"compio-log",
"compio-net",
"compio-runtime",
- "flume 0.12.0",
+ "flume",
"futures-util",
"libc",
"quinn-proto",
@@ -5329,17 +5359,6 @@ dependencies = [
"num-traits",
]
-[[package]]
-name = "flume"
-version = "0.11.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "da0e4dd2a88388a1f4ccc7c9ce104604dab68d9f408dc34cd45823d5a9069095"
-dependencies = [
- "futures-core",
- "futures-sink",
- "spin",
-]
-
[[package]]
name = "flume"
version = "0.12.0"
@@ -5759,6 +5778,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2eecf2d5dc9b66b732b97707a0210906b1d30523eb773193ab777c0c84b3e8d5"
dependencies = [
"polyval 0.7.3",
+ "zeroize",
]
[[package]]
@@ -7069,7 +7089,7 @@ checksum = "cd62e6b5e86ea8eeeb8db1de02880a6abc01a397b2ebb64b5d74ac255318f5cb"
[[package]]
name = "iggy"
-version = "0.11.1-edge.1"
+version = "0.11.1-edge.2"
dependencies = [
"async-broadcast",
"async-dropper",
@@ -7078,7 +7098,7 @@ dependencies = [
"bytemuck",
"bytes",
"dashmap",
- "flume 0.12.0",
+ "flume",
"futures",
"futures-util",
"iggy_binary_protocol",
@@ -7092,6 +7112,7 @@ dependencies = [
"reqwest-middleware",
"reqwest-retry",
"reqwest-tracing",
+ "ring",
"rustls",
"secrecy",
"serde",
@@ -7106,7 +7127,7 @@ dependencies = [
[[package]]
name = "iggy-bench"
-version = "0.6.1-edge.1"
+version = "0.6.1-edge.2"
dependencies = [
"async-trait",
"bench-report",
@@ -7164,7 +7185,7 @@ dependencies = [
[[package]]
name = "iggy-cli"
-version = "0.14.1-edge.1"
+version = "0.14.1-edge.2"
dependencies = [
"anyhow",
"apple-native-keyring-store",
@@ -7198,7 +7219,7 @@ dependencies = [
[[package]]
name = "iggy-connectors"
-version = "0.5.1-edge.1"
+version = "0.5.1-edge.2"
dependencies = [
"apache-avro 0.22.0",
"async-trait",
@@ -7215,7 +7236,7 @@ dependencies = [
"dotenvy",
"figlet-rs",
"figment",
- "flume 0.12.0",
+ "flume",
"futures",
"iggy",
"iggy_common",
@@ -7298,7 +7319,7 @@ dependencies = [
[[package]]
name = "iggy-mcp"
-version = "0.5.1-edge.1"
+version = "0.5.1-edge.2"
dependencies = [
"axum",
"axum-server",
@@ -7333,12 +7354,13 @@ dependencies = [
[[package]]
name = "iggy_binary_protocol"
-version = "0.11.1-edge.1"
+version = "0.11.1-edge.2"
dependencies = [
"aligned-vec",
"bytemuck",
"bytes",
"enumset",
+ "ring",
"secrecy",
"serde",
"thiserror 2.0.21",
@@ -7347,9 +7369,10 @@ dependencies = [
[[package]]
name = "iggy_common"
-version = "0.11.1-edge.1"
+version = "0.11.1-edge.2"
dependencies = [
"aes-gcm 0.11.1",
+ "aes-gcm-siv",
"async-broadcast",
"async-trait",
"base64 0.23.1",
@@ -7361,6 +7384,7 @@ dependencies = [
"clap",
"comfy-table",
"derive_more",
+ "hex",
"humantime",
"iggy_binary_protocol",
"papaya",
@@ -7378,6 +7402,7 @@ dependencies = [
"ulid",
"url",
"uuid",
+ "xaes-256-gcm",
]
[[package]]
@@ -7666,7 +7691,8 @@ dependencies = [
"humantime",
"iggy_common",
"iggy_connector_sdk",
- "rumqttc",
+ "rumqttc-v4-next",
+ "rumqttc-v5-next",
"rustls-native-certs",
"rustls-pemfile",
"secrecy",
@@ -8120,7 +8146,8 @@ dependencies = [
"reqwest-retry",
"ring",
"rmcp",
- "rumqttc",
+ "rumqttc-v4-next",
+ "rumqttc-v5-next",
"rust-s3",
"secrecy",
"serde",
@@ -8532,7 +8559,7 @@ dependencies = [
"backon",
"cfg-if",
"event-listener",
- "flume 0.12.0",
+ "flume",
"futures-core",
"futures-io",
"tracing",
@@ -9409,6 +9436,16 @@ dependencies = [
"pxfm",
]
+[[package]]
+name = "mqttbytes-core-next"
+version = "0.34.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "366b6ba2b4209ca4bc5ac731ccddf570d09831981eed07e5fbd63564cf0cf1aa"
+dependencies = [
+ "bytes",
+ "thiserror 2.0.21",
+]
+
[[package]]
name = "multiversion_no_op"
version = "1.0.0"
@@ -10872,6 +10909,7 @@ dependencies = [
"cpubits",
"cpufeatures 0.3.1",
"universal-hash 0.6.1",
+ "zeroize",
]
[[package]]
@@ -12138,23 +12176,66 @@ dependencies = [
]
[[package]]
-name = "rumqttc"
-version = "0.25.1"
+name = "rumqttc-core-next"
+version = "0.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "0feff8d882bff0b2fddaf99355a10336d43dd3ed44204f85ece28cf9626ab519"
+checksum = "249896ab27ed630590971738264baa8f722f18965d2e387c706c40a3c2a572cc"
dependencies = [
+ "async-tungstenite",
+ "rustls-native-certs",
+ "rustls-pki-types",
+ "rustls-webpki",
+ "thiserror 2.0.21",
+ "tokio",
+ "tokio-rustls",
+]
+
+[[package]]
+name = "rumqttc-v4-next"
+version = "0.34.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b3767dde22a804ea806107d0f4331e33cb794d1b8468b11c86fb0c1366f07054"
+dependencies = [
+ "async-tungstenite",
"bytes",
"fixedbitset",
- "flume 0.11.1",
+ "flume",
"futures-util",
"log",
+ "mqttbytes-core-next",
+ "percent-encoding",
+ "rumqttc-core-next",
"rustls-native-certs",
- "rustls-pemfile",
- "rustls-webpki 0.102.8",
+ "rustls-pki-types",
+ "rustls-webpki",
+ "thiserror 2.0.21",
+ "tokio",
+ "tokio-rustls",
+ "tokio-util",
+ "url",
+]
+
+[[package]]
+name = "rumqttc-v5-next"
+version = "0.34.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3dfa6ddcc7a7dd5688f9bf78d8f81cb94f367bce56c055d8d94cf81ecb0518bf"
+dependencies = [
+ "async-tungstenite",
+ "bytes",
+ "fixedbitset",
+ "flume",
+ "futures-util",
+ "log",
+ "mqttbytes-core-next",
+ "percent-encoding",
+ "rumqttc-core-next",
+ "rustls-native-certs",
+ "rustls-pki-types",
+ "rustls-webpki",
"thiserror 2.0.21",
"tokio",
"tokio-rustls",
- "tokio-stream",
"tokio-util",
"url",
]
@@ -12328,7 +12409,7 @@ dependencies = [
"once_cell",
"ring",
"rustls-pki-types",
- "rustls-webpki 0.103.15",
+ "rustls-webpki",
"subtle",
"zeroize",
]
@@ -12345,7 +12426,7 @@ dependencies = [
"rustls",
"rustls-pki-types",
"rustls-platform-verifier",
- "rustls-webpki 0.103.15",
+ "rustls-webpki",
]
[[package]]
@@ -12393,7 +12474,7 @@ dependencies = [
"rustls",
"rustls-native-certs",
"rustls-platform-verifier-android",
- "rustls-webpki 0.103.15",
+ "rustls-webpki",
"security-framework",
"security-framework-sys",
"webpki-root-certs",
@@ -12406,17 +12487,6 @@ version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eec689c0bc40ff2458a5977b6619cb718087084a18e02a131c599b62d05e1a5f"
-[[package]]
-name = "rustls-webpki"
-version = "0.102.8"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "64ca1bc8749bd4cf37b5ce386cc146580777b4e8572c7b97baf22c83f444bee9"
-dependencies = [
- "ring",
- "rustls-pki-types",
- "untrusted 0.9.0",
-]
-
[[package]]
name = "rustls-webpki"
version = "0.103.15"
@@ -12912,7 +12982,7 @@ dependencies = [
[[package]]
name = "server"
-version = "0.9.1-edge.1"
+version = "0.9.1-edge.2"
dependencies = [
"ahash",
"arc-swap",
@@ -13607,7 +13677,7 @@ checksum = "488e99c397a62007e4229aec669a179816339afc6d2620ca6fa420dbee2e982c"
dependencies = [
"atoi",
"chrono",
- "flume 0.12.0",
+ "flume",
"form_urlencoded",
"futures-channel",
"futures-core",
@@ -16203,6 +16273,19 @@ dependencies = [
"time",
]
+[[package]]
+name = "xaes-256-gcm"
+version = "0.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bc0e35bc669c7ce781caf52f51a1e672237825014c7a8cbb1292e194800d97fe"
+dependencies = [
+ "aead 0.6.1",
+ "aead-stream",
+ "aes 0.9.3",
+ "aes-gcm 0.11.1",
+ "cipher 0.5.2",
+]
+
[[package]]
name = "xattr"
version = "1.6.1"
diff --git a/Cargo.toml b/Cargo.toml
index 96a1a62b5..d93a38834 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -88,7 +88,8 @@ warnings = "deny"
actix-cors = "0.7.2"
actix-files = "0.7.0"
actix-web = "4.16.0"
-aes-gcm = "0.11.1"
+aes-gcm = { version = "0.11.1", features = ["zeroize"] }
+aes-gcm-siv = { version = "0.12.1", features = ["zeroize"] }
ahash = { version = "0.8.12", features = ["serde"] }
aligned-vec = "0.6.4"
anyhow = "1.0.104"
@@ -212,11 +213,11 @@ hyper-util = { version = "0.1.21", features = ["server-auto", "service"] }
iceberg = "0.10.1"
iceberg-catalog-rest = "0.10.1"
iceberg-storage-opendal = "0.10.1"
-iggy = { path = "core/sdk", version = "0.11.1-edge.1" }
-iggy-cli = { path = "core/cli", version = "0.14.1-edge.1" }
+iggy = { path = "core/sdk", version = "0.11.1-edge.2" }
+iggy-cli = { path = "core/cli", version = "0.14.1-edge.2" }
iggy-gateway-kafka = { path = "gateways/kafka" }
-iggy_binary_protocol = { path = "core/binary_protocol", version = "0.11.1-edge.1" }
-iggy_common = { path = "core/common", version = "0.11.1-edge.1" }
+iggy_binary_protocol = { path = "core/binary_protocol", version = "0.11.1-edge.2" }
+iggy_common = { path = "core/common", version = "0.11.1-edge.2" }
iggy_connector_doris_sink = { path = "core/connectors/sinks/doris_sink" }
iggy_connector_sdk = { path = "core/connectors/sdk", version = "0.5.1-edge.2" }
indexmap = "2.14.2"
@@ -304,7 +305,8 @@ rmcp = "3.5.1"
rmp = "0.8.15"
rmp-serde = "1.3.1"
rolling-file = "0.2.0"
-rumqttc = { version = "0.25.1", features = ["url"] }
+rumqttc-v4 = { package = "rumqttc-v4-next", version = "0.34.0", features = ["url"] }
+rumqttc-v5 = { package = "rumqttc-v5-next", version = "0.34.0", features = ["url"] }
rust-embed = "8.13.0"
rust-s3 = { version = "0.38.0", default-features = false, features = ["tokio-rustls-tls", "tags"] }
rustls = { version = "0.23.46", features = ["ring"] }
@@ -396,6 +398,7 @@ webpki-roots = "1.0.9"
windows-native-keyring-store = "1.1.0"
wiremock = "0.6"
x509-parser = "0.18.1"
+xaes-256-gcm = "0.1.0"
yew = { version = "0.23", features = ["csr"] }
yew-router = "0.20"
zbus-secret-service-keyring-store = { version = "1.0.1", features = ["rt-async-io-crypto-rust"] }
diff --git a/README.md b/README.md
index 07990be32..a837c5827 100644
--- a/README.md
+++ b/README.md
@@ -104,7 +104,7 @@ The name is an abbreviation for the Italian Greyhound - small yet extremely fast
- **TLS** support for all transport protocols (TCP, WebSocket, QUIC, HTTPS)
- **[Connectors](https://github.com/apache/iggy/tree/master/core/connectors)** - sinks, sources and data transformations based on the **custom Rust plugins**
- **[Model Context Protocol](https://github.com/apache/iggy/tree/master/core/ai/mcp)** - provide context to LLM with **MCP server**
-- Optional server-side as well as client-side **data encryption** using AES-256-GCM
+- Optional server-side as well as client-side **data encryption** using AES-256-GCM, server-side with a separate key per stream
- Optional metadata support in the form of **message headers**
- Server-side message compression is not supported yet. Topic compression values
are reserved for future disk/network compression support; use message headers
@@ -146,6 +146,8 @@ The server, SDKs and CLI are versioned independently. For example, server 0.9.0
We also publish edge releases (versions with an `-edge.N` suffix, and `apache/iggy:edge`) for the SDKs and the Docker images. They track the latest changes, but are not guaranteed to be stable and are not recommended for production use.
+This checkout moves the binary protocol to 0.12.0 and the metadata snapshot format to 13. Both are breaking: older clients and servers do not interoperate with it, and data directories written by older releases are refused. See [Upgrade recovery](core/server/README.md#upgrade-recovery).
+
---
## Supported languages SDK
@@ -294,6 +296,8 @@ Rust HTTP callers can use `HttpClient::send_messages_with_durability` to read th
The CLI exposes `--durability persisted` and `--consumer-offset-durability persisted` on `topic create`. Select either independently. The policy names describe completion guarantees and do not prescribe an I/O syscall.
+Streams, topics and users carry options from one catalog, listed by `iggy options <scope>` and changed by one sparse update. Each encrypted stream has its own data key, wrapped under the server's `[encryption] key`, and a stream created without encryption stays plaintext for life. Rotating the server key is a two-phase change through `previous_keys` and `retired_kek_fingerprints`. See [Resource options](core/server/README.md#resource-options) and [Stream encryption](core/server/README.md [...]
+
For the detailed documentation of the configuration file, please refer to the [configuration](https://iggy.apache.org/docs/server/configuration) section.
---
diff --git a/bdd/cpp/features/step_definitions/messaging_steps.cpp b/bdd/cpp/features/step_definitions/messaging_steps.cpp
index ed894441d..cfe55991f 100644
--- a/bdd/cpp/features/step_definitions/messaging_steps.cpp
+++ b/bdd/cpp/features/step_definitions/messaging_steps.cpp
@@ -46,7 +46,7 @@ WHEN("^I create a stream with name \"([^\"]{1,255})\"$") {
REGEX_PARAM(std::string, name);
cucumber::ScenarioScope<bdd::GlobalContext> context;
- context->client->create_stream(name, error_info);
+ context->client->create_stream(name, {}, error_info);
}
THEN("^the stream should be created successfully$") {
diff --git a/bdd/cpp/features/step_definitions/resource_options_steps.cpp b/bdd/cpp/features/step_definitions/resource_options_steps.cpp
new file mode 100644
index 000000000..598fab9af
--- /dev/null
+++ b/bdd/cpp/features/step_definitions/resource_options_steps.cpp
@@ -0,0 +1,475 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// Unique step-class prefix so the generated CukeObject<n> symbols do not clash with the other
+// step files when linked together (see background_steps.cpp for details).
+#define CUKE_OBJECT_PREFIX IggyBddResourceOptions
+
+#include <gtest/gtest.h>
+
+#include <cucumber-cpp/autodetect.hpp>
+
+#include <cstddef>
+#include <cstdint>
+#include <limits>
+#include <optional>
+#include <stdexcept>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "world.hpp"
+
+namespace {
+
+constexpr std::size_t kKeyLength = 32;
+constexpr std::uint32_t kPartitionId = 0;
+constexpr std::size_t kFingerprintHex = 16;
+
+constexpr iggy::StreamOptionKey kStreamOptionKeys[] = {
+ iggy::StreamOptionKey::Name,
+ iggy::StreamOptionKey::EncryptionKey,
+ iggy::StreamOptionKey::EncryptionKeySource,
+ iggy::StreamOptionKey::EncryptionKeyFingerprint,
+ iggy::StreamOptionKey::EncryptionKekFingerprint,
+};
+
+constexpr iggy::TopicOptionKey kTopicOptionKeys[] = {
+ iggy::TopicOptionKey::Name,
+ iggy::TopicOptionKey::CompressionAlgorithm,
+ iggy::TopicOptionKey::MessageExpiry,
+ iggy::TopicOptionKey::MaxTopicSize,
+ iggy::TopicOptionKey::SegmentSize,
+ iggy::TopicOptionKey::Durability,
+ iggy::TopicOptionKey::ConsumerOffsetDurability,
+ iggy::TopicOptionKey::PartitionResizePolicy,
+ iggy::TopicOptionKey::MessagesRequiredToSave,
+ iggy::TopicOptionKey::SizeOfMessagesRequiredToSave,
+ iggy::TopicOptionKey::PreallocateSegments,
+};
+
+iggy::IggyBlockingClient &Sdk(bdd::GlobalContext &context) {
+ if (!context.sdk) {
+ auto client =
+ iggy::IggyBlockingClient::Builder()
+ .WithServerAddress(bdd::required_env("IGGY_TCP_ADDRESS"))
+ .WithAutoLogin(bdd::required_env("IGGY_ROOT_USERNAME"), bdd::required_env("IGGY_ROOT_PASSWORD"))
+ .Build();
+ client.Connect();
+ context.sdk.emplace(std::move(client));
+ }
+ return *context.sdk;
+}
+
+template <typename Key, std::size_t Count>
+Key KeyFromName(const Key (&keys)[Count], const std::string &name) {
+ for (const auto key : keys) {
+ if (iggy::to_string(key) == name) {
+ return key;
+ }
+ }
+ throw std::invalid_argument("Unknown option key " + name);
+}
+
+iggy::UserStatus UserStatusFromName(const std::string &name) {
+ if (name == "active") {
+ return iggy::UserStatus::Active;
+ }
+ if (name == "inactive") {
+ return iggy::UserStatus::Inactive;
+ }
+ throw std::invalid_argument("Unknown user status " + name);
+}
+
+iggy::Identifier StreamIdentifier(const bdd::GlobalContext &context) {
+ return iggy::Identifier::Numeric(context.last_stream_id.value());
+}
+
+iggy::Identifier TopicIdentifier(const bdd::GlobalContext &context) {
+ return iggy::Identifier::Numeric(context.last_topic_id.value());
+}
+
+iggy::StreamDetails FetchStream(bdd::GlobalContext &context) {
+ return Sdk(context).GetStream(StreamIdentifier(context));
+}
+
+iggy::TopicDetails FetchTopic(bdd::GlobalContext &context) {
+ return Sdk(context).GetTopic(StreamIdentifier(context), TopicIdentifier(context));
+}
+
+iggy::UserInfoDetails FetchUser(bdd::GlobalContext &context) {
+ return Sdk(context).GetUser(iggy::Identifier::Numeric(context.last_user_id.value()));
+}
+
+void CreateStream(bdd::GlobalContext &context,
+ const std::string &name,
+ const std::optional<std::vector<std::uint8_t>> &key) {
+ iggy::StreamCreateOptions options;
+ if (key) {
+ options.SetEncryptionKey(*key);
+ }
+ const auto stream = Sdk(context).CreateStream(name, options);
+ context.last_stream_id = stream.Id();
+ context.last_stream_key = key;
+}
+
+void CreateTopic(bdd::GlobalContext &context, const std::string &name, iggy::TopicCreateOptions options) {
+ const auto topic = Sdk(context).CreateTopic(StreamIdentifier(context), name, options);
+ context.last_topic_id = topic.Id();
+}
+
+iggy::ffi::HeaderField StringField(const std::string &text) {
+ iggy::ffi::HeaderField field;
+ field.kind = static_cast<std::uint8_t>(iggy::ffi::HeaderKind::String);
+ field.value = bdd::to_payload(text);
+ return field;
+}
+
+void SendMessage(bdd::GlobalContext &context, const std::string &payload, rust::Vec<iggy::ffi::HeaderEntry> headers) {
+ iggy::ffi::ErrorInfo error_info{};
+ rust::Vec<iggy::ffi::IggyMessageToSend> messages;
+ messages.push_back(iggy::ffi::make_message(bdd::to_payload(payload), std::move(headers)));
+ context.client->send_messages(bdd::make_numeric_identifier(context.last_stream_id.value()),
+ bdd::make_numeric_identifier(context.last_topic_id.value()), "partition_id",
+ bdd::partition_id_bytes(kPartitionId), std::move(messages), error_info);
+}
+
+iggy::ffi::IggyMessagePolled PollFirst(bdd::GlobalContext &context) {
+ iggy::ffi::ErrorInfo error_info{};
+ auto polled =
+ context.client->poll_messages(bdd::make_numeric_identifier(context.last_stream_id.value()),
+ bdd::make_numeric_identifier(context.last_topic_id.value()), kPartitionId,
+ "consumer", bdd::make_numeric_identifier(1), "offset", 0, 1, false, error_info);
+ if (polled.messages.empty()) {
+ throw std::runtime_error("No message was polled");
+ }
+ return std::move(polled.messages[0]);
+}
+
+const iggy::OptionSpec &StreamOptionSpec(const bdd::GlobalContext &context, const std::string &key) {
+ for (const auto &spec : context.last_option_specs) {
+ if (spec.Key() == key) {
+ return spec;
+ }
+ }
+ throw std::runtime_error("Stream option " + key + " was not described");
+}
+
+void UpdateTopic(bdd::GlobalContext &context, const iggy::TopicUpdateOptions &options) {
+ Sdk(context).UpdateTopic(StreamIdentifier(context), TopicIdentifier(context), options);
+}
+
+} // namespace
+
+WHEN("^I create a stream named \"([^\"]+)\" with an encryption key of 32 bytes valued ([0-9]+)$") {
+ REGEX_PARAM(std::string, name);
+ REGEX_PARAM(int, byte);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ CreateStream(*context, name, std::vector<std::uint8_t>(kKeyLength, static_cast<std::uint8_t>(byte)));
+}
+
+GIVEN("^a stream named \"([^\"]+)\" with an encryption key of 32 bytes valued ([0-9]+) exists$") {
+ REGEX_PARAM(std::string, name);
+ REGEX_PARAM(int, byte);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ CreateStream(*context, name, std::vector<std::uint8_t>(kKeyLength, static_cast<std::uint8_t>(byte)));
+}
+
+WHEN("^I create a stream named \"([^\"]+)\" without an encryption key$") {
+ REGEX_PARAM(std::string, name);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ CreateStream(*context, name, std::nullopt);
+}
+
+GIVEN("^a stream named \"([^\"]+)\" exists$") {
+ REGEX_PARAM(std::string, name);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ CreateStream(*context, name, std::nullopt);
+}
+
+THEN("^the stream should report encryption key source \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, source);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto stream = FetchStream(*context);
+ ASSERT_TRUE(stream.EncryptionKeySource().has_value());
+ EXPECT_EQ(iggy::to_string(*stream.EncryptionKeySource()), source);
+}
+
+THEN("^the stream encryption key fingerprint should match the key$") {
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ ASSERT_TRUE(context->last_stream_key.has_value());
+ const auto stream = FetchStream(*context);
+ EXPECT_EQ(stream.EncryptionKeyFingerprint(), iggy::EncryptionKeyFingerprint(*context->last_stream_key));
+}
+
+THEN("^the stream should report an encryption key fingerprint$") {
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto stream = FetchStream(*context);
+ ASSERT_TRUE(stream.EncryptionKeyFingerprint().has_value());
+ EXPECT_EQ(stream.EncryptionKeyFingerprint()->size(), kFingerprintHex);
+}
+
+THEN("^the stream should report at least one KEK fingerprint$") {
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ EXPECT_FALSE(FetchStream(*context).EncryptionKekFingerprints().empty());
+}
+
+THEN("^the stream should expose no encryption key material$") {
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto stream = FetchStream(*context);
+ for (const auto *secret : {"encryption_key", "encryption_key_wrapped"}) {
+ EXPECT_EQ(stream.Options().Explicit().count(secret), 0u) << secret << " must never be returned";
+ EXPECT_EQ(stream.Options().Derived().count(secret), 0u) << secret << " must never be returned";
+ }
+}
+
+WHEN("^I create a topic named \"([^\"]+)\" with ([0-9]+) partitions? in the stream$") {
+ REGEX_PARAM(std::string, name);
+ REGEX_PARAM(int, partitions_count);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ CreateTopic(*context, name,
+ iggy::TopicCreateOptions()
+ .SetPartitionsCount(static_cast<std::uint32_t>(partitions_count))
+ .SetMessageExpiry(iggy::Expiry::NeverExpire()));
+}
+
+WHEN("^I send a message with payload \"([^\"]+)\" to the topic$") {
+ REGEX_PARAM(std::string, payload);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ SendMessage(*context, payload, rust::Vec<iggy::ffi::HeaderEntry>());
+}
+
+WHEN("^I send a message with payload \"([^\"]+)\" and header \"([^\"]+)\" set to \"([^\"]+)\" to the topic$") {
+ REGEX_PARAM(std::string, payload);
+ REGEX_PARAM(std::string, header);
+ REGEX_PARAM(std::string, value);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ rust::Vec<iggy::ffi::HeaderEntry> headers;
+ iggy::ffi::HeaderEntry entry;
+ entry.key = StringField(header);
+ entry.value = StringField(value);
+ headers.push_back(std::move(entry));
+ SendMessage(*context, payload, std::move(headers));
+}
+
+THEN("^polling the topic should return the payload \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, payload);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ EXPECT_EQ(bdd::payload_to_string(PollFirst(*context).payload), payload);
+}
+
+THEN("^polling the topic should return the payload \"([^\"]+)\" and header \"([^\"]+)\" set to \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, payload);
+ REGEX_PARAM(std::string, header);
+ REGEX_PARAM(std::string, value);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto message = PollFirst(*context);
+ EXPECT_EQ(bdd::payload_to_string(message.payload), payload);
+ bool found = false;
+ for (const auto &entry : message.user_headers) {
+ if (entry.key.kind == static_cast<std::uint8_t>(iggy::ffi::HeaderKind::String) &&
+ bdd::payload_to_string(entry.key.value) == header) {
+ found = true;
+ EXPECT_EQ(entry.value.kind, static_cast<std::uint8_t>(iggy::ffi::HeaderKind::String));
+ EXPECT_EQ(bdd::payload_to_string(entry.value.value), value);
+ }
+ }
+ EXPECT_TRUE(found) << "header " << header << " is missing";
+}
+
+WHEN("^I rename the stream to \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, name);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ Sdk(*context).UpdateStream(StreamIdentifier(*context), iggy::StreamUpdateOptions().SetName(name));
+}
+
+THEN("^the stream name should be \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, name);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ EXPECT_EQ(FetchStream(*context).Name(), name);
+}
+
+WHEN("^I request a reset of the stream option \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, key);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto options = iggy::StreamUpdateOptions().Reset(KeyFromName(kStreamOptionKeys, key));
+ try {
+ Sdk(*context).UpdateStream(StreamIdentifier(*context), options);
+ context->last_update_rejected = false;
+ } catch (const iggy::IggyException &) {
+ context->last_update_rejected = true;
+ }
+}
+
+THEN("^the stream update should be rejected$") {
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ EXPECT_TRUE(context->last_update_rejected) << "the update must fail";
+}
+
+WHEN("^I describe the stream options$") {
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ context->last_option_specs = Sdk(*context).DescribeOptions(iggy::OptionsScope::Stream);
+}
+
+THEN("^the stream option \"([^\"]+)\" should be updatable and not secret$") {
+ REGEX_PARAM(std::string, key);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto &spec = StreamOptionSpec(*context, key);
+ EXPECT_TRUE(spec.Updatable() && !spec.Secret() && !spec.ReadOnly());
+}
+
+THEN("^the stream option \"([^\"]+)\" should be create only and secret$") {
+ REGEX_PARAM(std::string, key);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto &spec = StreamOptionSpec(*context, key);
+ EXPECT_TRUE(!spec.Updatable() && !spec.Resettable() && spec.Secret() && !spec.ReadOnly());
+}
+
+THEN("^the stream option \"([^\"]+)\" should be read only$") {
+ REGEX_PARAM(std::string, key);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto &spec = StreamOptionSpec(*context, key);
+ EXPECT_TRUE(spec.ReadOnly() && !spec.Updatable() && !spec.Secret());
+}
+
+GIVEN(
+ "^a topic named \"([^\"]+)\" exists in the stream with message expiry ([0-9]+) microseconds and max topic "
+ "size ([0-9]+) bytes$") {
+ REGEX_PARAM(std::string, name);
+ REGEX_PARAM(std::uint64_t, expiry);
+ REGEX_PARAM(std::uint64_t, max_size);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ CreateTopic(*context, name,
+ iggy::TopicCreateOptions()
+ .SetPartitionsCount(1)
+ .SetMessageExpiry(iggy::Expiry::Duration(expiry))
+ .SetMaxTopicSize(iggy::MaxTopicSize::FromBytes(max_size)));
+}
+
+WHEN("^I update the topic message expiry to ([0-9]+) microseconds without a name$") {
+ REGEX_PARAM(std::uint64_t, expiry);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ UpdateTopic(*context, iggy::TopicUpdateOptions().SetMessageExpiry(iggy::Expiry::Duration(expiry)));
+}
+
+WHEN("^I reset the topic option \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, key);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ UpdateTopic(*context, iggy::TopicUpdateOptions().Reset(KeyFromName(kTopicOptionKeys, key)));
+}
+
+WHEN("^I rename the topic to \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, name);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ UpdateTopic(*context, iggy::TopicUpdateOptions().SetName(name));
+}
+
+THEN("^the topic name should be \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, name);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ EXPECT_EQ(FetchTopic(*context).Name(), name);
+}
+
+THEN("^the topic message expiry should be ([0-9]+) microseconds set explicitly$") {
+ REGEX_PARAM(std::uint64_t, expiry);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto topic = FetchTopic(*context);
+ EXPECT_EQ(topic.MessageExpiry(), expiry);
+ EXPECT_EQ(topic.Options().Explicit().count(std::string(iggy::to_string(iggy::TopicOptionKey::MessageExpiry))), 1u);
+}
+
+THEN("^the topic max topic size should be ([0-9]+) bytes set explicitly$") {
+ REGEX_PARAM(std::uint64_t, max_size);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto topic = FetchTopic(*context);
+ EXPECT_EQ(topic.MaxTopicSize(), max_size);
+ EXPECT_EQ(topic.Options().Explicit().count(std::string(iggy::to_string(iggy::TopicOptionKey::MaxTopicSize))), 1u);
+}
+
+THEN("^the topic max topic size should be derived from the server default$") {
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto topic = FetchTopic(*context);
+ const auto key = std::string(iggy::to_string(iggy::TopicOptionKey::MaxTopicSize));
+ EXPECT_EQ(topic.Options().Explicit().count(key), 0u);
+ EXPECT_EQ(topic.Options().Derived().count(key), 1u);
+ EXPECT_EQ(topic.MaxTopicSize(), std::numeric_limits<std::uint64_t>::max());
+}
+
+GIVEN("^a user named \"([^\"]+)\" with password \"([^\"]+)\" exists$") {
+ REGEX_PARAM(std::string, username);
+ REGEX_PARAM(std::string, password);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ const auto user =
+ Sdk(*context).CreateUser(username, password, iggy::UserStatus::Active, std::nullopt, iggy::UserCreateOptions());
+ context->last_user_id = user.Id();
+}
+
+WHEN("^I update the user username to \"([^\"]+)\" and status to \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, username);
+ REGEX_PARAM(std::string, status);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ Sdk(*context).UpdateUser(iggy::Identifier::Numeric(context->last_user_id.value()),
+ iggy::UserUpdateOptions().SetUsername(username).SetStatus(UserStatusFromName(status)));
+}
+
+THEN("^the user should have username \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, username);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ EXPECT_EQ(FetchUser(*context).Username(), username);
+}
+
+THEN("^the user should have status \"([^\"]+)\"$") {
+ REGEX_PARAM(std::string, status);
+ cucumber::ScenarioScope<bdd::GlobalContext> context;
+
+ EXPECT_EQ(FetchUser(*context).Status(), UserStatusFromName(status));
+}
diff --git a/bdd/cpp/features/step_definitions/world.hpp b/bdd/cpp/features/step_definitions/world.hpp
index 886be4fe4..a10214778 100644
--- a/bdd/cpp/features/step_definitions/world.hpp
+++ b/bdd/cpp/features/step_definitions/world.hpp
@@ -21,10 +21,12 @@
#include <cstdint>
#include <cstdlib>
+#include <optional>
#include <stdexcept>
#include <string>
#include <vector>
+#include "iggy.hpp"
#include "lib.rs.h"
namespace bdd {
@@ -49,6 +51,14 @@ struct GlobalContext {
PolledData polled;
std::vector<std::uint8_t> raw_response;
std::string raw_error;
+ // Public client for the option steps, connected on first use.
+ std::optional<iggy::IggyBlockingClient> sdk;
+ std::optional<std::uint32_t> last_stream_id;
+ std::optional<std::uint32_t> last_topic_id;
+ std::optional<std::uint32_t> last_user_id;
+ std::optional<std::vector<std::uint8_t>> last_stream_key;
+ bool last_update_rejected = false;
+ std::vector<iggy::OptionSpec> last_option_specs;
std::string call_error;
GlobalContext() = default;
@@ -69,7 +79,7 @@ struct GlobalContext {
inline std::string required_env(const char *name) {
const char *value = std::getenv(name);
if (value == nullptr || *value == '\0') {
- throw std::runtime_error(std::string(name) + " must be set; run the suite via scripts/run-bdd-tests.sh");
+ throw std::runtime_error(std::string(name) + " must be set. Run the suite via scripts/run-bdd-tests.sh");
}
return std::string(value);
}
diff --git a/bdd/cpp/scripts/entrypoint.sh b/bdd/cpp/scripts/entrypoint.sh
index 94c4eed4b..f04fcfd4e 100755
--- a/bdd/cpp/scripts/entrypoint.sh
+++ b/bdd/cpp/scripts/entrypoint.sh
@@ -58,6 +58,8 @@ tags=()
case "${BDD_FEATURE:-all}" in
basic_messaging) tags=(--tags @basic-messaging) ;;
raw_command) tags=(--tags @raw-command) ;;
+ stream_options) tags=(--tags @stream-options) ;;
+ resource_options_patch) tags=(--tags @resource-options-patch) ;;
session_contract) tags=(--tags @session-contract) ;;
esac
diff --git a/bdd/docker-compose.coverage.yml b/bdd/docker-compose.coverage.yml
index 7caa2153b..1bdf7614c 100644
--- a/bdd/docker-compose.coverage.yml
+++ b/bdd/docker-compose.coverage.yml
@@ -39,7 +39,7 @@ services:
trap 'cargo llvm-cov report --lcov --output-path /reports/rust-bdd-coverage.lcov' EXIT
TEST_ARGS=()
case "$$BDD_FEATURE" in
- basic_messaging|leader_redirection|raw_command|session_contract|stream_crud) TEST_ARGS=(--test "$$BDD_FEATURE") ;;
+ basic_messaging|leader_redirection|raw_command|session_contract|stream_crud|stream_options|resource_options_patch) TEST_ARGS=(--test "$$BDD_FEATURE") ;;
esac
cargo test --locked -p bdd --features "$$BDD_RUST_FEATURES" "$${TEST_ARGS[@]}"
@@ -103,6 +103,8 @@ services:
case "$$BDD_FEATURE" in
basic_messaging) FILTER='--filter-trait Category=basic-messaging' ;;
leader_redirection) FILTER='--filter-trait Category=requires-leader-awareness' ;;
+ stream_options) FILTER='--filter-trait Category=stream-options' ;;
+ resource_options_patch) FILTER='--filter-trait Category=resource-options-patch' ;;
session_contract) FILTER='--filter-trait Category=session-contract' ;;
esac
dotnet test $$FILTER --coverage --coverage-output-format cobertura --coverage-output csharp-bdd-coverage.cobertura.xml --results-directory /reports
@@ -120,5 +122,7 @@ services:
raw_command) export CUCUMBER_FILTER_TAGS='@raw-command' ;;
session_contract) export CUCUMBER_FILTER_TAGS='@session-contract' ;;
stream_crud) export CUCUMBER_FILTER_TAGS='@stream-crud' ;;
+ stream_options) export CUCUMBER_FILTER_TAGS='@stream-options' ;;
+ resource_options_patch) export CUCUMBER_FILTER_TAGS='@resource-options-patch' ;;
esac
gradle --no-daemon test jacocoTestReport && cp build/reports/jacoco/test/jacocoTestReport.xml /reports/java-bdd-coverage.xml
diff --git a/bdd/docker-compose.server.yml b/bdd/docker-compose.server.yml
index dcda9539e..0f4c30093 100644
--- a/bdd/docker-compose.server.yml
+++ b/bdd/docker-compose.server.yml
@@ -70,6 +70,9 @@ services:
- IGGY_HTTP_ADDRESS=0.0.0.0:3000
- IGGY_QUIC_ADDRESS=0.0.0.0:8080
- IGGY_WEBSOCKET_ADDRESS=0.0.0.0:8070
+ # Every feature shares this server, and stream_options asserts server-generated stream keys.
+ - IGGY_ENCRYPTION_ENABLED=true
+ - IGGY_ENCRYPTION_KEY=/rvT1xP4V8u1EAhk4xDdqzqM2UOPXyy9XYkl4uRShgE=
volumes:
- iggy_data:/app/local_data
- ../target:/profiles
diff --git a/bdd/docker-compose.yml b/bdd/docker-compose.yml
index 3dd3e1049..681f689c4 100644
--- a/bdd/docker-compose.yml
+++ b/bdd/docker-compose.yml
@@ -36,6 +36,8 @@ services:
- ./scenarios/raw_command.feature:/app/features/raw_command.feature
- ./scenarios/session_contract.feature:/app/features/session_contract.feature
- ./scenarios/stream_crud.feature:/app/features/stream_crud.feature
+ - ./scenarios/stream_options.feature:/app/features/stream_options.feature
+ - ./scenarios/resource_options_patch.feature:/app/features/resource_options_patch.feature
command:
- sh
- -c
@@ -46,6 +48,8 @@ services:
raw_command) cargo test -p bdd --features "$$BDD_RUST_FEATURES" --test raw_command ;;
session_contract) cargo test -p bdd --features "$$BDD_RUST_FEATURES" --test session_contract ;;
stream_crud) cargo test -p bdd --features "$$BDD_RUST_FEATURES" --test stream_crud ;;
+ stream_options) cargo test -p bdd --features "$$BDD_RUST_FEATURES" --test stream_options ;;
+ resource_options_patch) cargo test -p bdd --features "$$BDD_RUST_FEATURES" --test resource_options_patch ;;
*) cargo test -p bdd --features "$$BDD_RUST_FEATURES" ;;
esac
networks:
@@ -63,6 +67,8 @@ services:
volumes:
- ./scenarios/basic_messaging.feature:/app/features/basic_messaging.feature
- ./scenarios/raw_command.feature:/app/features/raw_command.feature
+ - ./scenarios/stream_options.feature:/app/features/stream_options.feature
+ - ./scenarios/resource_options_patch.feature:/app/features/resource_options_patch.feature
- ./scenarios/session_contract.feature:/app/features/session_contract.feature
working_dir: /app
command:
@@ -70,10 +76,12 @@ services:
- -c
- |
case "$$BDD_FEATURE" in
- basic_messaging) TEST_PATH='tests/test_basic_messaging.py' ;;
- raw_command) TEST_PATH='tests/test_raw_command.py' ;;
- session_contract) TEST_PATH='tests/test_session_contract.py' ;;
- *) TEST_PATH='tests/' ;;
+ basic_messaging) TEST_PATH='tests/test_basic_messaging.py' ;;
+ raw_command) TEST_PATH='tests/test_raw_command.py' ;;
+ session_contract) TEST_PATH='tests/test_session_contract.py' ;;
+ stream_options) TEST_PATH='tests/test_stream_options.py' ;;
+ resource_options_patch) TEST_PATH='tests/test_resource_options_patch.py' ;;
+ *) TEST_PATH='tests/' ;;
esac
uv run --project /workspace/bdd/python pytest $$TEST_PATH -v
networks:
@@ -90,6 +98,7 @@ services:
volumes:
- ./scenarios/basic_messaging.feature:/app/features/basic_messaging.feature
- ./scenarios/raw_command.feature:/app/features/raw_command.feature
+ - ./scenarios/stream_options.feature:/app/features/stream_options.feature
- ./scenarios/session_contract.feature:/app/features/session_contract.feature
command:
- sh
@@ -99,6 +108,7 @@ services:
basic_messaging) GROUP='--group basic-messaging' ;;
raw_command) GROUP='--group raw-command' ;;
session_contract) GROUP='--group session-contract' ;;
+ stream_options) GROUP='--group stream-options' ;;
*) GROUP='' ;;
esac
./scripts/test.sh --configuration /workspace/bdd/php/phpunit.xml.dist $$GROUP
@@ -133,10 +143,12 @@ services:
- -c
- |
case "$$BDD_FEATURE" in
- basic_messaging) TAGS='--tags @basic-messaging' ;;
- raw_command) TAGS='--tags @raw-command' ;;
- session_contract) TAGS='--tags @session-contract' ;;
- *) TAGS='' ;;
+ basic_messaging) TAGS='--tags @basic-messaging' ;;
+ raw_command) TAGS='--tags @raw-command' ;;
+ session_contract) TAGS='--tags @session-contract' ;;
+ stream_options) TAGS='--tags @stream-options' ;;
+ resource_options_patch) TAGS='--tags @resource-options-patch' ;;
+ *) TAGS='' ;;
esac
npm run test:bdd -- $$TAGS
networks:
@@ -160,6 +172,8 @@ services:
basic_messaging) FILTER='--filter-trait Category=basic-messaging' ;;
leader_redirection) FILTER='--filter-trait Category=requires-leader-awareness' ;;
raw_command) FILTER='--filter-trait Category=raw-command' ;;
+ stream_options) FILTER='--filter-trait Category=stream-options' ;;
+ resource_options_patch) FILTER='--filter-trait Category=resource-options-patch' ;;
session_contract) FILTER='--filter-trait Category=session-contract' ;;
esac
dotnet test $$FILTER
@@ -181,6 +195,8 @@ services:
- ./scenarios/raw_command.feature:/app/features/raw_command.feature
- ./scenarios/session_contract.feature:/app/features/session_contract.feature
- ./scenarios/stream_crud.feature:/app/features/stream_crud.feature
+ - ./scenarios/stream_options.feature:/app/features/stream_options.feature
+ - ./scenarios/resource_options_patch.feature:/app/features/resource_options_patch.feature
command:
- sh
- -c
@@ -191,6 +207,8 @@ services:
raw_command) export CUCUMBER_FILTER_TAGS='@raw-command' ;;
session_contract) export CUCUMBER_FILTER_TAGS='@session-contract' ;;
stream_crud) export CUCUMBER_FILTER_TAGS='@stream-crud' ;;
+ stream_options) export CUCUMBER_FILTER_TAGS='@stream-options' ;;
+ resource_options_patch) export CUCUMBER_FILTER_TAGS='@resource-options-patch' ;;
esac
gradle --no-daemon test
networks:
@@ -208,6 +226,8 @@ services:
volumes:
- ./scenarios/basic_messaging.feature:/app/features/basic_messaging.feature
- ./scenarios/raw_command.feature:/app/features/raw_command.feature
+ - ./scenarios/stream_options.feature:/app/features/stream_options.feature
+ - ./scenarios/resource_options_patch.feature:/app/features/resource_options_patch.feature
- ./scenarios/session_contract.feature:/app/features/session_contract.feature
networks:
- iggy-bdd-network
diff --git a/bdd/go/tests/basic_messaging.go b/bdd/go/tests/basic_messaging.go
index 21ad3e12b..ef91b206a 100644
--- a/bdd/go/tests/basic_messaging.go
+++ b/bdd/go/tests/basic_messaging.go
@@ -228,7 +228,7 @@ func (s basicMessagingSteps) givenNoStreams(ctx context.Context) error {
func (s basicMessagingSteps) whenCreateStream(ctx context.Context, streamName string) error {
c := getBasicMessagingCtx(ctx)
- stream, err := c.client.CreateStream(ctx, streamName)
+ stream, err := c.client.CreateStream(ctx, streamName, nil)
if err != nil {
return fmt.Errorf("failed to create stream: %w", err)
}
@@ -259,14 +259,12 @@ func (s basicMessagingSteps) whenCreateTopic(ctx context.Context,
partitionsCount uint32) error {
c := getBasicMessagingCtx(ctx)
streamIdentifier, _ := iggcon.NewIdentifier(streamID)
+ expiry := iggcon.IggyExpiryNeverExpire
topic, err := c.client.CreateTopic(
ctx,
streamIdentifier,
topicName,
- partitionsCount,
- iggcon.CompressionAlgorithmNone,
- iggcon.IggyExpiryNeverExpire,
- 0,
+ &iggcon.TopicOptions{PartitionsCount: &partitionsCount, MessageExpiry: &expiry},
)
if err != nil {
return fmt.Errorf("failed to create topic: %w", err)
diff --git a/bdd/go/tests/leader_redirection.go b/bdd/go/tests/leader_redirection.go
index 8593f5512..6e8efe105 100644
--- a/bdd/go/tests/leader_redirection.go
+++ b/bdd/go/tests/leader_redirection.go
@@ -334,7 +334,7 @@ func (s leaderSteps) whenCreateStream(ctx context.Context, streamName string) er
return errors.New("client should be available")
}
- stream, err := cli.CreateStream(ctx, streamName)
+ stream, err := cli.CreateStream(ctx, streamName, nil)
if err != nil {
return err
}
diff --git a/bdd/go/tests/resource_options.go b/bdd/go/tests/resource_options.go
new file mode 100644
index 000000000..9a250ba22
--- /dev/null
+++ b/bdd/go/tests/resource_options.go
@@ -0,0 +1,607 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package tests
+
+import (
+ "bytes"
+ "context"
+ "encoding/base64"
+ "encoding/hex"
+ "errors"
+ "fmt"
+ "slices"
+
+ "github.com/apache/iggy/bdd/go/tests/env"
+ "github.com/apache/iggy/foreign/go/client"
+ "github.com/apache/iggy/foreign/go/client/tcp"
+ iggcon "github.com/apache/iggy/foreign/go/contracts"
+ "github.com/cucumber/godog"
+)
+
+// resourceOptionsPartition is the partition every message of these scenarios
+// goes to and is polled from.
+const resourceOptionsPartition = 0
+
+type resourceOptionsCtxKey struct{}
+
+type resourceOptionsCtx struct {
+ client iggcon.Client
+ streamId *uint32
+ streamKey []byte
+ topicId *uint32
+ userId *uint32
+ createdStreams []uint32
+ createdUsers []uint32
+ updateError error
+ optionSpecs []iggcon.OptionSpec
+}
+
+func getResourceOptionsCtx(ctx context.Context) *resourceOptionsCtx {
+ return ctx.Value(resourceOptionsCtxKey{}).(*resourceOptionsCtx)
+}
+
+type resourceOptionsSteps struct{}
+
+func (resourceOptionsSteps) givenRunningServer(context.Context) error {
+ return nil
+}
+
+func (resourceOptionsSteps) givenAuthenticationAsRoot(ctx context.Context) error {
+ state := getResourceOptionsCtx(ctx)
+ iggyClient, err := client.NewIggyClient(client.WithTcp(tcp.WithServerAddress(env.ServerAddress())))
+ if err != nil {
+ return fmt.Errorf("create client: %w", err)
+ }
+ if err = iggyClient.Connect(ctx); err != nil {
+ return fmt.Errorf("connect client: %w", err)
+ }
+ username, password := env.RootCredentials()
+ if _, err = iggyClient.LoginUser(ctx, username, password); err != nil {
+ return fmt.Errorf("authenticate client: %w", err)
+ }
+ state.client = iggyClient
+ return nil
+}
+
+func (s resourceOptionsSteps) whenCreateKeyedStream(ctx context.Context, name string, value int) error {
+ state := getResourceOptionsCtx(ctx)
+ key := bytes.Repeat([]byte{byte(value)}, iggcon.StreamEncryptionKeyLength)
+ if err := s.createStream(ctx, name, &iggcon.StreamOptions{EncryptionKey: key}); err != nil {
+ return err
+ }
+ state.streamKey = key
+ return nil
+}
+
+func (s resourceOptionsSteps) whenCreateKeylessStream(ctx context.Context, name string) error {
+ getResourceOptionsCtx(ctx).streamKey = nil
+ return s.createStream(ctx, name, nil)
+}
+
+func (resourceOptionsSteps) createStream(ctx context.Context, name string, options *iggcon.StreamOptions) error {
+ state := getResourceOptionsCtx(ctx)
+ stream, err := state.client.CreateStream(ctx, name, options)
+ if err != nil {
+ return fmt.Errorf("create stream %s: %w", name, err)
+ }
+ state.streamId = &stream.Id
+ state.createdStreams = append(state.createdStreams, stream.Id)
+ return nil
+}
+
+func (s resourceOptionsSteps) thenStreamSource(ctx context.Context, want string) error {
+ stream, err := s.fetchStream(ctx)
+ if err != nil {
+ return err
+ }
+ source, ok := stream.EncryptionKeySource()
+ if !ok || source != iggcon.StreamEncryptionSource(want) {
+ return fmt.Errorf("encryption key source is %q (reported %t), want %q", source, ok, want)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenFingerprintMatchesKey(ctx context.Context) error {
+ stream, err := s.fetchStream(ctx)
+ if err != nil {
+ return err
+ }
+ want, err := iggcon.EncryptionKeyFingerprint(getResourceOptionsCtx(ctx).streamKey)
+ if err != nil {
+ return fmt.Errorf("fingerprint the stream key: %w", err)
+ }
+ if got, ok := stream.EncryptionKeyFingerprint(); !ok || got != want {
+ return fmt.Errorf("encryption key fingerprint is %q (reported %t), want %q", got, ok, want)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenStreamReportsFingerprint(ctx context.Context) error {
+ stream, err := s.fetchStream(ctx)
+ if err != nil {
+ return err
+ }
+ if fingerprint, ok := stream.EncryptionKeyFingerprint(); !ok || len(fingerprint) != 16 {
+ return fmt.Errorf("encryption key fingerprint is %q (reported %t), want 16 hex characters", fingerprint, ok)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenStreamReportsKek(ctx context.Context) error {
+ stream, err := s.fetchStream(ctx)
+ if err != nil {
+ return err
+ }
+ if len(stream.EncryptionKekFingerprints()) == 0 {
+ return errors.New("the stream reports no KEK fingerprint")
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenNoKeyMaterial(ctx context.Context) error {
+ stream, err := s.fetchStream(ctx)
+ if err != nil {
+ return err
+ }
+ key := getResourceOptionsCtx(ctx).streamKey
+ for _, options := range []map[string]iggcon.HeaderValue{stream.Options, stream.DerivedOptions} {
+ for _, secret := range []string{string(iggcon.StreamOptionEncryptionKey), "encryption_key_wrapped", "critical_keys"} {
+ if _, found := options[secret]; found {
+ return fmt.Errorf("%s must never be returned", secret)
+ }
+ }
+ if key == nil {
+ continue
+ }
+ for name, value := range options {
+ if bytes.Contains(value.Value, key) ||
+ bytes.Contains(value.Value, []byte(base64.StdEncoding.EncodeToString(key))) ||
+ bytes.Contains(value.Value, []byte(hex.EncodeToString(key))) {
+ return fmt.Errorf("option %s carries the stream key", name)
+ }
+ }
+ }
+ return nil
+}
+
+func (resourceOptionsSteps) whenCreateTopic(ctx context.Context, name string, partitions uint32) error {
+ state := getResourceOptionsCtx(ctx)
+ expiry := iggcon.IggyExpiryNeverExpire
+ topic, err := state.client.CreateTopic(ctx, streamIdentifier(state), name,
+ &iggcon.TopicOptions{PartitionsCount: &partitions, MessageExpiry: &expiry})
+ if err != nil {
+ return fmt.Errorf("create topic %s: %w", name, err)
+ }
+ state.topicId = &topic.Id
+ return nil
+}
+
+func (s resourceOptionsSteps) whenSendPayload(ctx context.Context, payload string) error {
+ return s.sendMessage(ctx, payload, nil)
+}
+
+func (s resourceOptionsSteps) whenSendPayloadWithHeader(ctx context.Context, payload, header, value string) error {
+ return s.sendMessage(ctx, payload, []iggcon.HeaderEntry{{
+ Key: iggcon.HeaderKey{Kind: iggcon.String, Value: []byte(header)},
+ Value: iggcon.HeaderValue{Kind: iggcon.String, Value: []byte(value)},
+ }})
+}
+
+func (resourceOptionsSteps) sendMessage(ctx context.Context, payload string, headers []iggcon.HeaderEntry) error {
+ state := getResourceOptionsCtx(ctx)
+ var options []iggcon.IggyMessageOpt
+ if len(headers) > 0 {
+ options = append(options, iggcon.WithUserHeaders(headers))
+ }
+ message, err := iggcon.NewIggyMessage([]byte(payload), options...)
+ if err != nil {
+ return fmt.Errorf("build message: %w", err)
+ }
+ _, err = state.client.SendMessages(ctx, streamIdentifier(state), topicIdentifier(state),
+ iggcon.PartitionId(resourceOptionsPartition), []iggcon.IggyMessage{message})
+ if err != nil {
+ return fmt.Errorf("send message: %w", err)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenPollReturnsPayload(ctx context.Context, payload string) error {
+ message, err := s.pollFirst(ctx)
+ if err != nil {
+ return err
+ }
+ if string(message.Payload) != payload {
+ return fmt.Errorf("polled payload %q, want %q", message.Payload, payload)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenPollReturnsPayloadAndHeader(ctx context.Context, payload, header, value string) error {
+ message, err := s.pollFirst(ctx)
+ if err != nil {
+ return err
+ }
+ if string(message.Payload) != payload {
+ return fmt.Errorf("polled payload %q, want %q", message.Payload, payload)
+ }
+ headers, err := iggcon.DeserializeHeaders(message.UserHeaders)
+ if err != nil {
+ return fmt.Errorf("read user headers: %w", err)
+ }
+ want := iggcon.HeaderEntry{
+ Key: iggcon.HeaderKey{Kind: iggcon.String, Value: []byte(header)},
+ Value: iggcon.HeaderValue{Kind: iggcon.String, Value: []byte(value)},
+ }
+ if !slices.ContainsFunc(headers, func(entry iggcon.HeaderEntry) bool {
+ return entry.Key.Kind == want.Key.Kind && bytes.Equal(entry.Key.Value, want.Key.Value) &&
+ entry.Value.Kind == want.Value.Kind && bytes.Equal(entry.Value.Value, want.Value.Value)
+ }) {
+ return fmt.Errorf("polled headers %v do not carry %s=%s", headers, header, value)
+ }
+ return nil
+}
+
+func (resourceOptionsSteps) pollFirst(ctx context.Context) (iggcon.IggyMessage, error) {
+ state := getResourceOptionsCtx(ctx)
+ partition := uint32(resourceOptionsPartition)
+ polled, err := state.client.PollMessages(ctx, streamIdentifier(state), topicIdentifier(state),
+ iggcon.DefaultConsumer(), iggcon.OffsetPollingStrategy(0), 1, false, &partition)
+ if err != nil {
+ return iggcon.IggyMessage{}, fmt.Errorf("poll messages: %w", err)
+ }
+ if len(polled.Messages) == 0 {
+ return iggcon.IggyMessage{}, errors.New("the poll returned no message")
+ }
+ return polled.Messages[0], nil
+}
+
+func (resourceOptionsSteps) whenRenameStream(ctx context.Context, name string) error {
+ state := getResourceOptionsCtx(ctx)
+ if err := state.client.UpdateStream(ctx, streamIdentifier(state), iggcon.StreamUpdate{Name: &name}); err != nil {
+ return fmt.Errorf("rename stream: %w", err)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenStreamName(ctx context.Context, name string) error {
+ stream, err := s.fetchStream(ctx)
+ if err != nil {
+ return err
+ }
+ if stream.Name != name {
+ return fmt.Errorf("stream name is %q, want %q", stream.Name, name)
+ }
+ return nil
+}
+
+func (resourceOptionsSteps) whenResetStreamOption(ctx context.Context, key string) error {
+ state := getResourceOptionsCtx(ctx)
+ state.updateError = state.client.UpdateStream(ctx, streamIdentifier(state),
+ iggcon.StreamUpdate{Reset: []iggcon.StreamOptionKey{iggcon.StreamOptionKey(key)}})
+ return nil
+}
+
+func (resourceOptionsSteps) thenStreamUpdateRejected(ctx context.Context) error {
+ if getResourceOptionsCtx(ctx).updateError == nil {
+ return errors.New("the update must fail")
+ }
+ return nil
+}
+
+func (resourceOptionsSteps) whenDescribeStreamOptions(ctx context.Context) error {
+ state := getResourceOptionsCtx(ctx)
+ specs, err := state.client.DescribeOptions(ctx, iggcon.OptionsScopeStream)
+ if err != nil {
+ return fmt.Errorf("describe options: %w", err)
+ }
+ state.optionSpecs = specs
+ return nil
+}
+
+func (s resourceOptionsSteps) thenOptionUpdatable(ctx context.Context, key string) error {
+ spec, err := s.optionSpec(ctx, key)
+ if err != nil {
+ return err
+ }
+ if !spec.Updatable || spec.Secret || spec.ReadOnly {
+ return fmt.Errorf("%s is %+v, want updatable and not secret", key, spec)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenOptionCreateOnlySecret(ctx context.Context, key string) error {
+ spec, err := s.optionSpec(ctx, key)
+ if err != nil {
+ return err
+ }
+ if spec.Updatable || spec.Resettable || !spec.Secret || spec.ReadOnly {
+ return fmt.Errorf("%s is %+v, want create only and secret", key, spec)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenOptionReadOnly(ctx context.Context, key string) error {
+ spec, err := s.optionSpec(ctx, key)
+ if err != nil {
+ return err
+ }
+ if !spec.ReadOnly || spec.Updatable || spec.Secret {
+ return fmt.Errorf("%s is %+v, want read only", key, spec)
+ }
+ return nil
+}
+
+func (resourceOptionsSteps) optionSpec(ctx context.Context, key string) (iggcon.OptionSpec, error) {
+ for _, spec := range getResourceOptionsCtx(ctx).optionSpecs {
+ if spec.Key == key {
+ return spec, nil
+ }
+ }
+ return iggcon.OptionSpec{}, fmt.Errorf("the catalog does not describe %s", key)
+}
+
+func (resourceOptionsSteps) givenTopicWithSettings(ctx context.Context, name string, expiry, maxTopicSize uint64) error {
+ state := getResourceOptionsCtx(ctx)
+ partitions := uint32(1)
+ messageExpiry := iggcon.Duration(expiry)
+ topic, err := state.client.CreateTopic(ctx, streamIdentifier(state), name, &iggcon.TopicOptions{
+ PartitionsCount: &partitions,
+ MessageExpiry: &messageExpiry,
+ MaxTopicSize: &maxTopicSize,
+ })
+ if err != nil {
+ return fmt.Errorf("create topic %s: %w", name, err)
+ }
+ state.topicId = &topic.Id
+ return nil
+}
+
+func (s resourceOptionsSteps) whenUpdateTopicExpiry(ctx context.Context, expiry uint64) error {
+ messageExpiry := iggcon.Duration(expiry)
+ return s.updateTopic(ctx, iggcon.TopicUpdate{MessageExpiry: &messageExpiry})
+}
+
+func (s resourceOptionsSteps) whenResetTopicOption(ctx context.Context, key string) error {
+ return s.updateTopic(ctx, iggcon.TopicUpdate{Reset: []iggcon.TopicOptionKey{iggcon.TopicOptionKey(key)}})
+}
+
+func (s resourceOptionsSteps) whenRenameTopic(ctx context.Context, name string) error {
+ return s.updateTopic(ctx, iggcon.TopicUpdate{Name: &name})
+}
+
+func (resourceOptionsSteps) updateTopic(ctx context.Context, update iggcon.TopicUpdate) error {
+ state := getResourceOptionsCtx(ctx)
+ if err := state.client.UpdateTopic(ctx, streamIdentifier(state), topicIdentifier(state), update); err != nil {
+ return fmt.Errorf("update topic: %w", err)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenTopicName(ctx context.Context, name string) error {
+ topic, err := s.fetchTopic(ctx)
+ if err != nil {
+ return err
+ }
+ if topic.Name != name {
+ return fmt.Errorf("topic name is %q, want %q", topic.Name, name)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenTopicExpiryExplicit(ctx context.Context, expiry uint64) error {
+ topic, err := s.fetchTopic(ctx)
+ if err != nil {
+ return err
+ }
+ if uint64(topic.MessageExpiry) != expiry {
+ return fmt.Errorf("message expiry is %d, want %d", topic.MessageExpiry, expiry)
+ }
+ if _, explicit := topic.Options[string(iggcon.TopicOptionMessageExpiry)]; !explicit {
+ return errors.New("message expiry must be set explicitly")
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenTopicMaxSizeExplicit(ctx context.Context, maxTopicSize uint64) error {
+ topic, err := s.fetchTopic(ctx)
+ if err != nil {
+ return err
+ }
+ if topic.MaxTopicSize != maxTopicSize {
+ return fmt.Errorf("max topic size is %d, want %d", topic.MaxTopicSize, maxTopicSize)
+ }
+ if _, explicit := topic.Options[string(iggcon.TopicOptionMaxTopicSize)]; !explicit {
+ return errors.New("max topic size must be set explicitly")
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenTopicMaxSizeDerived(ctx context.Context) error {
+ topic, err := s.fetchTopic(ctx)
+ if err != nil {
+ return err
+ }
+ if _, explicit := topic.Options[string(iggcon.TopicOptionMaxTopicSize)]; explicit {
+ return errors.New("max topic size must no longer be explicit")
+ }
+ if _, derived := topic.DerivedOptions[string(iggcon.TopicOptionMaxTopicSize)]; !derived {
+ return errors.New("max topic size must be derived from the server default")
+ }
+ return nil
+}
+
+func (resourceOptionsSteps) givenUser(ctx context.Context, username, password string) error {
+ state := getResourceOptionsCtx(ctx)
+ user, err := state.client.CreateUser(ctx, username, password, iggcon.Active, nil, &iggcon.UserOptions{})
+ if err != nil {
+ return fmt.Errorf("create user %s: %w", username, err)
+ }
+ state.userId = &user.Id
+ state.createdUsers = append(state.createdUsers, user.Id)
+ return nil
+}
+
+func (resourceOptionsSteps) whenUpdateUser(ctx context.Context, username, status string) error {
+ state := getResourceOptionsCtx(ctx)
+ userStatus, err := parseUserStatus(status)
+ if err != nil {
+ return err
+ }
+ update := iggcon.UserUpdate{Username: &username, Status: &userStatus}
+ if err = state.client.UpdateUser(ctx, userIdentifier(state), update); err != nil {
+ return fmt.Errorf("update user: %w", err)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenUserUsername(ctx context.Context, username string) error {
+ user, err := s.fetchUser(ctx)
+ if err != nil {
+ return err
+ }
+ if user.Username != username {
+ return fmt.Errorf("username is %q, want %q", user.Username, username)
+ }
+ return nil
+}
+
+func (s resourceOptionsSteps) thenUserStatus(ctx context.Context, status string) error {
+ user, err := s.fetchUser(ctx)
+ if err != nil {
+ return err
+ }
+ want, err := parseUserStatus(status)
+ if err != nil {
+ return err
+ }
+ if user.Status != want {
+ return fmt.Errorf("user status is %d, want %s", user.Status, status)
+ }
+ return nil
+}
+
+func (resourceOptionsSteps) fetchStream(ctx context.Context) (*iggcon.StreamDetails, error) {
+ state := getResourceOptionsCtx(ctx)
+ stream, err := state.client.GetStream(ctx, streamIdentifier(state))
+ if err != nil {
+ return nil, fmt.Errorf("get stream: %w", err)
+ }
+ return stream, nil
+}
+
+func (resourceOptionsSteps) fetchTopic(ctx context.Context) (*iggcon.TopicDetails, error) {
+ state := getResourceOptionsCtx(ctx)
+ topic, err := state.client.GetTopic(ctx, streamIdentifier(state), topicIdentifier(state))
+ if err != nil {
+ return nil, fmt.Errorf("get topic: %w", err)
+ }
+ return topic, nil
+}
+
+func (resourceOptionsSteps) fetchUser(ctx context.Context) (*iggcon.UserInfoDetails, error) {
+ state := getResourceOptionsCtx(ctx)
+ user, err := state.client.GetUser(ctx, userIdentifier(state))
+ if err != nil {
+ return nil, fmt.Errorf("get user: %w", err)
+ }
+ return user, nil
+}
+
+func initResourceOptionsScenario(sc *godog.ScenarioContext) {
+ sc.Before(func(context.Context, *godog.Scenario) (context.Context, error) {
+ return context.WithValue(context.Background(), resourceOptionsCtxKey{}, &resourceOptionsCtx{}), nil
+ })
+ s := resourceOptionsSteps{}
+ sc.Step(`^I have a running Iggy server$`, s.givenRunningServer)
+ sc.Step(`^I am authenticated as the root user$`, s.givenAuthenticationAsRoot)
+ sc.Step(`^I create a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+)$`, s.whenCreateKeyedStream)
+ sc.Step(`^a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+) exists$`, s.whenCreateKeyedStream)
+ sc.Step(`^I create a stream named "([^"]+)" without an encryption key$`, s.whenCreateKeylessStream)
+ sc.Step(`^a stream named "([^"]+)" exists$`, s.whenCreateKeylessStream)
+ sc.Step(`^the stream should report encryption key source "([^"]+)"$`, s.thenStreamSource)
+ sc.Step(`^the stream encryption key fingerprint should match the key$`, s.thenFingerprintMatchesKey)
+ sc.Step(`^the stream should report an encryption key fingerprint$`, s.thenStreamReportsFingerprint)
+ sc.Step(`^the stream should report at least one KEK fingerprint$`, s.thenStreamReportsKek)
+ sc.Step(`^the stream should expose no encryption key material$`, s.thenNoKeyMaterial)
+ sc.Step(`^I create a topic named "([^"]+)" with (\d+) partitions? in the stream$`, s.whenCreateTopic)
+ sc.Step(`^I send a message with payload "([^"]+)" to the topic$`, s.whenSendPayload)
+ sc.Step(`^I send a message with payload "([^"]+)" and header "([^"]+)" set to "([^"]+)" to the topic$`, s.whenSendPayloadWithHeader)
+ sc.Step(`^polling the topic should return the payload "([^"]+)"$`, s.thenPollReturnsPayload)
+ sc.Step(`^polling the topic should return the payload "([^"]+)" and header "([^"]+)" set to "([^"]+)"$`, s.thenPollReturnsPayloadAndHeader)
+ sc.Step(`^I rename the stream to "([^"]+)"$`, s.whenRenameStream)
+ sc.Step(`^the stream name should be "([^"]+)"$`, s.thenStreamName)
+ sc.Step(`^I request a reset of the stream option "([^"]+)"$`, s.whenResetStreamOption)
+ sc.Step(`^the stream update should be rejected$`, s.thenStreamUpdateRejected)
+ sc.Step(`^I describe the stream options$`, s.whenDescribeStreamOptions)
+ sc.Step(`^the stream option "([^"]+)" should be updatable and not secret$`, s.thenOptionUpdatable)
+ sc.Step(`^the stream option "([^"]+)" should be create only and secret$`, s.thenOptionCreateOnlySecret)
+ sc.Step(`^the stream option "([^"]+)" should be read only$`, s.thenOptionReadOnly)
+ sc.Step(`^a topic named "([^"]+)" exists in the stream with message expiry (\d+) microseconds and max topic size (\d+) bytes$`, s.givenTopicWithSettings)
+ sc.Step(`^I update the topic message expiry to (\d+) microseconds without a name$`, s.whenUpdateTopicExpiry)
+ sc.Step(`^I reset the topic option "([^"]+)"$`, s.whenResetTopicOption)
+ sc.Step(`^I rename the topic to "([^"]+)"$`, s.whenRenameTopic)
+ sc.Step(`^the topic name should be "([^"]+)"$`, s.thenTopicName)
+ sc.Step(`^the topic message expiry should be (\d+) microseconds set explicitly$`, s.thenTopicExpiryExplicit)
+ sc.Step(`^the topic max topic size should be (\d+) bytes set explicitly$`, s.thenTopicMaxSizeExplicit)
+ sc.Step(`^the topic max topic size should be derived from the server default$`, s.thenTopicMaxSizeDerived)
+ sc.Step(`^a user named "([^"]+)" with password "([^"]+)" exists$`, s.givenUser)
+ sc.Step(`^I update the user username to "([^"]+)" and status to "([^"]+)"$`, s.whenUpdateUser)
+ sc.Step(`^the user should have username "([^"]+)"$`, s.thenUserUsername)
+ sc.Step(`^the user should have status "([^"]+)"$`, s.thenUserStatus)
+ sc.After(func(ctx context.Context, _ *godog.Scenario, scenarioError error) (context.Context, error) {
+ state := getResourceOptionsCtx(ctx)
+ if state.client == nil {
+ return ctx, scenarioError
+ }
+ for _, id := range state.createdStreams {
+ streamId, err := iggcon.NewIdentifier(id)
+ scenarioError = errors.Join(scenarioError, err, state.client.DeleteStream(ctx, streamId))
+ }
+ for _, id := range state.createdUsers {
+ userId, err := iggcon.NewIdentifier(id)
+ scenarioError = errors.Join(scenarioError, err, state.client.DeleteUser(ctx, userId))
+ }
+ return ctx, errors.Join(scenarioError, state.client.Close())
+ })
+}
+
+func streamIdentifier(state *resourceOptionsCtx) iggcon.Identifier {
+ identifier, _ := iggcon.NewIdentifier(*state.streamId)
+ return identifier
+}
+
+func topicIdentifier(state *resourceOptionsCtx) iggcon.Identifier {
+ identifier, _ := iggcon.NewIdentifier(*state.topicId)
+ return identifier
+}
+
+func userIdentifier(state *resourceOptionsCtx) iggcon.Identifier {
+ identifier, _ := iggcon.NewIdentifier(*state.userId)
+ return identifier
+}
+
+func parseUserStatus(status string) (iggcon.UserStatus, error) {
+ switch status {
+ case "active":
+ return iggcon.Active, nil
+ case "inactive":
+ return iggcon.Inactive, nil
+ default:
+ return 0, fmt.Errorf("unknown user status %q", status)
+ }
+}
diff --git a/bdd/go/tests/suite_test.go b/bdd/go/tests/suite_test.go
index 7841899a7..3adf9c1e8 100644
--- a/bdd/go/tests/suite_test.go
+++ b/bdd/go/tests/suite_test.go
@@ -76,6 +76,29 @@ func TestFeatures(t *testing.T) {
})
}
+ if feature == "all" || feature == "stream_options" {
+ suites = append(suites, godog.TestSuite{
+ ScenarioInitializer: initResourceOptionsScenario,
+ Options: &godog.Options{
+ Format: "pretty",
+ Paths: []string{"../../scenarios/stream_options.feature"},
+ Strict: true,
+ TestingT: t,
+ },
+ })
+ }
+ if feature == "all" || feature == "resource_options_patch" {
+ suites = append(suites, godog.TestSuite{
+ ScenarioInitializer: initResourceOptionsScenario,
+ Options: &godog.Options{
+ Format: "pretty",
+ Paths: []string{"../../scenarios/resource_options_patch.feature"},
+ Strict: true,
+ TestingT: t,
+ },
+ })
+ }
+
if len(suites) == 0 {
t.Fatalf("unknown BDD_FEATURE=%q", feature)
}
diff --git a/bdd/go/tests/tcp_test/stream_feature_create.go b/bdd/go/tests/tcp_test/stream_feature_create.go
index 5502c7967..4271f95b1 100644
--- a/bdd/go/tests/tcp_test/stream_feature_create.go
+++ b/bdd/go/tests/tcp_test/stream_feature_create.go
@@ -30,7 +30,7 @@ var _ = ginkgo.Describe("CREATE STREAM:", func() {
client := createAuthorizedConnection()
name := createRandomString(32)
- stream, err := client.CreateStream(context.Background(), name)
+ stream, err := client.CreateStream(context.Background(), name, nil)
defer deleteStreamAfterTests(stream.Id, client)
itShouldNotReturnError(err)
@@ -41,13 +41,13 @@ var _ = ginkgo.Describe("CREATE STREAM:", func() {
client := createAuthorizedConnection()
name := createRandomString(32)
- stream, err := client.CreateStream(context.Background(), name)
+ stream, err := client.CreateStream(context.Background(), name, nil)
defer deleteStreamAfterTests(stream.Id, client)
itShouldNotReturnError(err)
itShouldSuccessfullyCreateStream(stream.Id, name, client)
- _, err = client.CreateStream(context.Background(), name)
+ _, err = client.CreateStream(context.Background(), name, nil)
itShouldReturnSpecificError(err, ierror.ErrStreamNameAlreadyExists)
})
@@ -56,7 +56,7 @@ var _ = ginkgo.Describe("CREATE STREAM:", func() {
client := createAuthorizedConnection()
name := createRandomString(256)
- _, err := client.CreateStream(context.Background(), name)
+ _, err := client.CreateStream(context.Background(), name, nil)
itShouldReturnSpecificError(err, ierror.ErrInvalidStreamName)
})
@@ -65,7 +65,7 @@ var _ = ginkgo.Describe("CREATE STREAM:", func() {
ginkgo.When("User is not logged in", func() {
ginkgo.Context("and tries to create stream", func() {
client := createClient()
- _, err := client.CreateStream(context.Background(), createRandomString(32))
+ _, err := client.CreateStream(context.Background(), createRandomString(32), nil)
itShouldReturnUnauthenticatedError(err)
})
diff --git a/bdd/go/tests/tcp_test/stream_feature_get_by_id.go b/bdd/go/tests/tcp_test/stream_feature_get_by_id.go
index 71d4cc497..f4a001b05 100644
--- a/bdd/go/tests/tcp_test/stream_feature_get_by_id.go
+++ b/bdd/go/tests/tcp_test/stream_feature_get_by_id.go
@@ -58,21 +58,13 @@ var _ = ginkgo.Describe("GET STREAM BY ID:", func() {
// create two topics
t1Name := createRandomString(32)
t2Name := createRandomString(32)
- t1, err := client.CreateTopic(context.Background(), streamIdentifier,
- t1Name,
- 2,
- iggcon.CompressionAlgorithmNone,
- iggcon.Millisecond,
- math.MaxUint64)
+ partitions := uint32(2)
+ expiry := iggcon.Millisecond
+ maxTopicSize := uint64(math.MaxUint64)
+ options := &iggcon.TopicOptions{PartitionsCount: &partitions, MessageExpiry: &expiry, MaxTopicSize: &maxTopicSize}
+ t1, err := client.CreateTopic(context.Background(), streamIdentifier, t1Name, options)
itShouldNotReturnError(err)
- t2, err := client.CreateTopic(
- context.Background(),
- streamIdentifier,
- t2Name,
- 2,
- iggcon.CompressionAlgorithmNone,
- iggcon.Millisecond,
- math.MaxUint64)
+ t2, err := client.CreateTopic(context.Background(), streamIdentifier, t2Name, options)
itShouldNotReturnError(err)
itShouldSuccessfullyCreateTopic(streamId, t1.Id, t1Name, client)
itShouldSuccessfullyCreateTopic(streamId, t2.Id, t2Name, client)
diff --git a/bdd/go/tests/tcp_test/stream_feature_update.go b/bdd/go/tests/tcp_test/stream_feature_update.go
index d8ea7a5dd..c20b37904 100644
--- a/bdd/go/tests/tcp_test/stream_feature_update.go
+++ b/bdd/go/tests/tcp_test/stream_feature_update.go
@@ -34,7 +34,7 @@ var _ = ginkgo.Describe("UPDATE STREAM:", func() {
defer deleteStreamAfterTests(streamId, client)
newName := createRandomString(128)
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
- err := client.UpdateStream(context.Background(), streamIdentifier, newName)
+ err := client.UpdateStream(context.Background(), streamIdentifier, iggcon.StreamUpdate{Name: &newName})
itShouldNotReturnError(err)
itShouldSuccessfullyUpdateStream(streamId, newName, client)
})
@@ -47,14 +47,15 @@ var _ = ginkgo.Describe("UPDATE STREAM:", func() {
defer deleteStreamAfterTests(stream2Id, client)
stream2Identifier, _ := iggcon.NewIdentifier(stream2Id)
- err := client.UpdateStream(context.Background(), stream2Identifier, stream1Name)
+ err := client.UpdateStream(context.Background(), stream2Identifier, iggcon.StreamUpdate{Name: &stream1Name})
itShouldReturnSpecificError(err, ierror.ErrStreamNameAlreadyExists)
})
ginkgo.Context("and tries to update non-existing stream", func() {
client := createAuthorizedConnection()
- err := client.UpdateStream(context.Background(), randomU32Identifier(), createRandomString(128))
+ name := createRandomString(128)
+ err := client.UpdateStream(context.Background(), randomU32Identifier(), iggcon.StreamUpdate{Name: &name})
itShouldReturnSpecificError(err, ierror.ErrStreamIdNotFound)
})
@@ -64,7 +65,8 @@ var _ = ginkgo.Describe("UPDATE STREAM:", func() {
streamId, _ := successfullyCreateStream(prefix, client)
defer deleteStreamAfterTests(streamId, createAuthorizedConnection())
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
- err := client.UpdateStream(context.Background(), streamIdentifier, createRandomString(256))
+ name := createRandomString(256)
+ err := client.UpdateStream(context.Background(), streamIdentifier, iggcon.StreamUpdate{Name: &name})
itShouldReturnSpecificError(err, ierror.ErrInvalidStreamName)
})
@@ -73,7 +75,8 @@ var _ = ginkgo.Describe("UPDATE STREAM:", func() {
ginkgo.When("User is not logged in", func() {
ginkgo.Context("and tries to update stream", func() {
client := createClient()
- err := client.UpdateStream(context.Background(), randomU32Identifier(), createRandomString(128))
+ name := createRandomString(128)
+ err := client.UpdateStream(context.Background(), randomU32Identifier(), iggcon.StreamUpdate{Name: &name})
itShouldReturnUnauthenticatedError(err)
})
diff --git a/bdd/go/tests/tcp_test/stream_steps.go b/bdd/go/tests/tcp_test/stream_steps.go
index 2789bf4fb..47d1ea225 100644
--- a/bdd/go/tests/tcp_test/stream_steps.go
+++ b/bdd/go/tests/tcp_test/stream_steps.go
@@ -32,7 +32,7 @@ import (
func successfullyCreateStream(prefix string, client iggcon.Client) (uint32, string) {
name := createRandomStringWithPrefix(prefix, 128)
- stream, err := client.CreateStream(context.Background(), name)
+ stream, err := client.CreateStream(context.Background(), name, nil)
itShouldNotReturnError(err)
itShouldSuccessfullyCreateStream(stream.Id, name, client)
diff --git a/bdd/go/tests/tcp_test/topic_feature_create.go b/bdd/go/tests/tcp_test/topic_feature_create.go
index 902abcd2b..07b4beb79 100644
--- a/bdd/go/tests/tcp_test/topic_feature_create.go
+++ b/bdd/go/tests/tcp_test/topic_feature_create.go
@@ -35,14 +35,14 @@ var _ = ginkgo.Describe("CREATE TOPIC:", func() {
name := createRandomString(32)
defer deleteStreamAfterTests(streamId, client)
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
+ partitions := uint32(2)
+ expiry := iggcon.Millisecond
+ maxTopicSize := uint64(math.MaxUint64)
_, err := client.CreateTopic(
context.Background(),
streamIdentifier,
name,
- 2,
- iggcon.CompressionAlgorithmNone,
- iggcon.Millisecond,
- math.MaxUint64)
+ &iggcon.TopicOptions{PartitionsCount: &partitions, MessageExpiry: &expiry, MaxTopicSize: &maxTopicSize})
itShouldNotReturnError(err)
})
@@ -52,14 +52,14 @@ var _ = ginkgo.Describe("CREATE TOPIC:", func() {
streamId := createRandomUInt32()
name := createRandomString(32)
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
+ partitions := uint32(2)
+ expiry := iggcon.Millisecond
+ maxTopicSize := uint64(math.MaxUint64)
_, err := client.CreateTopic(
context.Background(),
streamIdentifier,
name,
- 2,
- iggcon.CompressionAlgorithmNone,
- iggcon.Millisecond,
- math.MaxUint64)
+ &iggcon.TopicOptions{PartitionsCount: &partitions, MessageExpiry: &expiry, MaxTopicSize: &maxTopicSize})
itShouldReturnSpecificError(err, ierror.ErrStreamIdNotFound)
})
@@ -71,14 +71,13 @@ var _ = ginkgo.Describe("CREATE TOPIC:", func() {
_, name := successfullyCreateTopic(streamId, client)
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
+ partitions := uint32(2)
+ maxTopicSize := uint64(math.MaxUint64)
_, err := client.CreateTopic(
context.Background(),
streamIdentifier,
name,
- 2,
- iggcon.CompressionAlgorithmNone,
- iggcon.IggyExpiryServerDefault,
- math.MaxUint64)
+ &iggcon.TopicOptions{PartitionsCount: &partitions, MaxTopicSize: &maxTopicSize})
itShouldReturnSpecificError(err, ierror.ErrTopicNameAlreadyExists)
})
@@ -88,14 +87,13 @@ var _ = ginkgo.Describe("CREATE TOPIC:", func() {
defer deleteStreamAfterTests(streamId, createAuthorizedConnection())
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
+ partitions := uint32(2)
+ maxTopicSize := uint64(math.MaxUint64)
_, err := client.CreateTopic(
context.Background(),
streamIdentifier,
createRandomString(256),
- 2,
- iggcon.CompressionAlgorithmNone,
- iggcon.IggyExpiryServerDefault,
- math.MaxUint64)
+ &iggcon.TopicOptions{PartitionsCount: &partitions, MaxTopicSize: &maxTopicSize})
itShouldReturnSpecificError(err, ierror.ErrInvalidTopicName)
})
@@ -105,14 +103,13 @@ var _ = ginkgo.Describe("CREATE TOPIC:", func() {
ginkgo.Context("and tries to create topic", func() {
client := createClient()
streamIdentifier, _ := iggcon.NewIdentifier[uint32](10)
+ partitions := uint32(2)
+ maxTopicSize := uint64(math.MaxUint64)
_, err := client.CreateTopic(
context.Background(),
streamIdentifier,
"name",
- 2,
- iggcon.CompressionAlgorithmNone,
- iggcon.IggyExpiryServerDefault,
- math.MaxUint64)
+ &iggcon.TopicOptions{PartitionsCount: &partitions, MaxTopicSize: &maxTopicSize})
itShouldReturnUnauthenticatedError(err)
})
diff --git a/bdd/go/tests/tcp_test/topic_feature_update.go b/bdd/go/tests/tcp_test/topic_feature_update.go
index 292883021..d375d11d2 100644
--- a/bdd/go/tests/tcp_test/topic_feature_update.go
+++ b/bdd/go/tests/tcp_test/topic_feature_update.go
@@ -37,14 +37,13 @@ var _ = ginkgo.Describe("UPDATE TOPIC:", func() {
newName := createRandomString(128)
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
topicIdentifier, _ := iggcon.NewIdentifier(topicId)
+ expiry := iggcon.Microsecond
+ maxTopicSize := uint64(math.MaxUint64)
err := client.UpdateTopic(
context.Background(),
streamIdentifier,
topicIdentifier,
- newName,
- iggcon.CompressionAlgorithmNone,
- iggcon.Microsecond,
- math.MaxUint64)
+ iggcon.TopicUpdate{Name: &newName, MessageExpiry: &expiry, MaxTopicSize: &maxTopicSize})
itShouldNotReturnError(err)
itShouldSuccessfullyUpdateTopic(streamId, topicId, newName, client)
})
@@ -57,28 +56,25 @@ var _ = ginkgo.Describe("UPDATE TOPIC:", func() {
topic2Id, _ := successfullyCreateTopic(streamId, client)
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
topic2Identifier, _ := iggcon.NewIdentifier(topic2Id)
+ maxTopicSize := uint64(math.MaxUint64)
err := client.UpdateTopic(
context.Background(),
streamIdentifier,
topic2Identifier,
- topic1Name,
- iggcon.CompressionAlgorithmNone,
- iggcon.IggyExpiryServerDefault,
- math.MaxUint64)
+ iggcon.TopicUpdate{Name: &topic1Name, MaxTopicSize: &maxTopicSize})
itShouldReturnSpecificError(err, ierror.ErrTopicNameAlreadyExists)
})
ginkgo.Context("and tries to update non-existing topic", func() {
client := createAuthorizedConnection()
+ name := createRandomString(128)
+ maxTopicSize := uint64(math.MaxUint64)
err := client.UpdateTopic(
context.Background(),
randomU32Identifier(),
randomU32Identifier(),
- createRandomString(128),
- 1,
- 0,
- math.MaxUint64)
+ iggcon.TopicUpdate{Name: &name, MaxTopicSize: &maxTopicSize})
itShouldReturnSpecificError(err, ierror.ErrStreamIdNotFound)
})
@@ -88,14 +84,13 @@ var _ = ginkgo.Describe("UPDATE TOPIC:", func() {
streamId, _ := successfullyCreateStream(prefix, client)
defer deleteStreamAfterTests(streamId, createAuthorizedConnection())
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
+ name := createRandomString(128)
+ maxTopicSize := uint64(math.MaxUint64)
err := client.UpdateTopic(
context.Background(),
streamIdentifier,
randomU32Identifier(),
- createRandomString(128),
- 1,
- 0,
- math.MaxUint64)
+ iggcon.TopicUpdate{Name: &name, MaxTopicSize: &maxTopicSize})
itShouldReturnSpecificError(err, ierror.ErrTopicIdNotFound)
})
@@ -107,14 +102,13 @@ var _ = ginkgo.Describe("UPDATE TOPIC:", func() {
topicId, _ := successfullyCreateTopic(streamId, client)
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
topicIdentifier, _ := iggcon.NewIdentifier(topicId)
+ name := createRandomString(256)
+ maxTopicSize := uint64(math.MaxUint64)
err := client.UpdateTopic(
context.Background(),
streamIdentifier,
topicIdentifier,
- createRandomString(256),
- 1,
- 0,
- math.MaxUint64)
+ iggcon.TopicUpdate{Name: &name, MaxTopicSize: &maxTopicSize})
itShouldReturnSpecificError(err, ierror.ErrInvalidTopicName)
})
@@ -123,7 +117,8 @@ var _ = ginkgo.Describe("UPDATE TOPIC:", func() {
ginkgo.When("User is not logged in", func() {
ginkgo.Context("and tries to update stream", func() {
client := createClient()
- err := client.UpdateStream(context.Background(), randomU32Identifier(), createRandomString(128))
+ name := createRandomString(128)
+ err := client.UpdateStream(context.Background(), randomU32Identifier(), iggcon.StreamUpdate{Name: &name})
itShouldReturnUnauthenticatedError(err)
})
diff --git a/bdd/go/tests/tcp_test/topic_steps.go b/bdd/go/tests/tcp_test/topic_steps.go
index a359f688d..d1715c489 100644
--- a/bdd/go/tests/tcp_test/topic_steps.go
+++ b/bdd/go/tests/tcp_test/topic_steps.go
@@ -33,14 +33,13 @@ import (
func successfullyCreateTopic(streamId uint32, client iggcon.Client) (uint32, string) {
name := createRandomString(128)
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
+ partitions := uint32(2)
+ maxTopicSize := uint64(math.MaxUint64)
topic, err := client.CreateTopic(
context.Background(),
streamIdentifier,
name,
- 2,
- 1,
- 0,
- math.MaxUint64)
+ &iggcon.TopicOptions{PartitionsCount: &partitions, MaxTopicSize: &maxTopicSize})
itShouldSuccessfullyCreateTopic(streamId, topic.Id, name, client)
itShouldNotReturnError(err)
diff --git a/bdd/go/tests/tcp_test/users_feature_create.go b/bdd/go/tests/tcp_test/users_feature_create.go
index 033267c3d..523d4f867 100644
--- a/bdd/go/tests/tcp_test/users_feature_create.go
+++ b/bdd/go/tests/tcp_test/users_feature_create.go
@@ -48,7 +48,7 @@ var _ = ginkgo.Describe("CREATE USER:", func() {
PollMessages: true,
SendMessages: true,
},
- })
+ }, nil)
identifier, _ := iggcon.NewIdentifier(username)
defer deleteUserAfterTests(identifier, client)
@@ -104,7 +104,7 @@ var _ = ginkgo.Describe("CREATE USER:", func() {
SendMessages: true,
},
Streams: userStreamPermissions,
- })
+ }, nil)
identifier, _ := iggcon.NewIdentifier(username)
defer deleteUserAfterTests(identifier, client)
defer deleteStreamAfterTests(streamId, client)
@@ -137,7 +137,7 @@ var _ = ginkgo.Describe("CREATE USER:", func() {
PollMessages: true,
SendMessages: true,
},
- })
+ }, nil)
itShouldReturnUnauthenticatedError(err)
})
diff --git a/bdd/go/tests/tcp_test/users_feature_password.go b/bdd/go/tests/tcp_test/users_feature_password.go
index 78d756235..7280395bc 100644
--- a/bdd/go/tests/tcp_test/users_feature_password.go
+++ b/bdd/go/tests/tcp_test/users_feature_password.go
@@ -49,7 +49,7 @@ var _ = ginkgo.Describe("CHANGE PASSWORD:", func() {
PollMessages: true,
SendMessages: true,
},
- })
+ }, nil)
itShouldNotReturnError(err)
identifier, _ := iggcon.NewIdentifier(username)
defer deleteUserAfterTests(identifier, client)
diff --git a/bdd/go/tests/tcp_test/users_feature_permissions.go b/bdd/go/tests/tcp_test/users_feature_permissions.go
index 55b734ca8..e70f762cc 100644
--- a/bdd/go/tests/tcp_test/users_feature_permissions.go
+++ b/bdd/go/tests/tcp_test/users_feature_permissions.go
@@ -62,8 +62,7 @@ var _ = ginkgo.Describe("UPDATE USER PERMISSIONS:", func() {
err := client.UpdateUser(
context.Background(),
randomU32Identifier(),
- &username,
- nil,
+ iggcon.UserUpdate{Username: &username},
)
itShouldReturnUnauthenticatedError(err)
})
diff --git a/bdd/go/tests/tcp_test/users_feature_update.go b/bdd/go/tests/tcp_test/users_feature_update.go
index 2a6df4cba..3b50bb1cd 100644
--- a/bdd/go/tests/tcp_test/users_feature_update.go
+++ b/bdd/go/tests/tcp_test/users_feature_update.go
@@ -36,8 +36,7 @@ var _ = ginkgo.Describe("UPDATE USER:", func() {
err := client.UpdateUser(
context.Background(),
identifier,
- &username,
- nil,
+ iggcon.UserUpdate{Username: &username},
)
itShouldNotReturnError(err)
@@ -53,8 +52,7 @@ var _ = ginkgo.Describe("UPDATE USER:", func() {
err := client.UpdateUser(
context.Background(),
randomU32Identifier(),
- &username,
- nil,
+ iggcon.UserUpdate{Username: &username},
)
itShouldReturnUnauthenticatedError(err)
})
diff --git a/bdd/go/tests/tcp_test/users_steps.go b/bdd/go/tests/tcp_test/users_steps.go
index 22a71dc99..908806c7d 100644
--- a/bdd/go/tests/tcp_test/users_steps.go
+++ b/bdd/go/tests/tcp_test/users_steps.go
@@ -47,7 +47,7 @@ func successfullyCreateUser(name string, client iggcon.Client) uint32 {
PollMessages: true,
SendMessages: true,
},
- })
+ }, nil)
itShouldNotReturnError(err)
nameIdentifier, _ := iggcon.NewIdentifier(name)
user, err := client.GetUser(context.Background(), nameIdentifier)
diff --git a/bdd/java/src/test/java/org/apache/iggy/bdd/BasicMessagingSteps.java b/bdd/java/src/test/java/org/apache/iggy/bdd/BasicMessagingSteps.java
index 34b800829..823ec2d13 100644
--- a/bdd/java/src/test/java/org/apache/iggy/bdd/BasicMessagingSteps.java
+++ b/bdd/java/src/test/java/org/apache/iggy/bdd/BasicMessagingSteps.java
@@ -31,7 +31,7 @@ import org.apache.iggy.message.Partitioning;
import org.apache.iggy.message.PollingStrategy;
import org.apache.iggy.stream.StreamBase;
import org.apache.iggy.stream.StreamDetails;
-import org.apache.iggy.topic.CompressionAlgorithm;
+import org.apache.iggy.stream.StreamUpdate;
import org.apache.iggy.topic.TopicDetails;
import java.math.BigInteger;
@@ -126,7 +126,11 @@ public class BasicMessagingSteps {
@When("I update the stream name to {string}")
public void updateStreamName(String streamName) {
- getClient().streams().updateStream(context.lastStreamId, streamName);
+ getClient()
+ .streams()
+ .updateStream(
+ context.lastStreamId,
+ StreamUpdate.builder().name(streamName).build());
}
@Then("getting the stream by its numeric ID should return name {string}")
@@ -156,15 +160,7 @@ public class BasicMessagingSteps {
@When("I create a topic with name {string} in stream {int} with {int} partitions")
public void createTopic(String topicName, int streamId, int partitions) {
- TopicDetails topic = getClient()
- .topics()
- .createTopic(
- (long) streamId,
- (long) partitions,
- CompressionAlgorithm.None,
- BigInteger.ZERO,
- BigInteger.ZERO,
- topicName);
+ TopicDetails topic = getClient().topics().createTopic((long) streamId, topicName, (long) partitions);
context.lastTopicId = topic.id();
context.lastTopicName = topic.name();
diff --git a/bdd/java/src/test/java/org/apache/iggy/bdd/ResourceOptionsSteps.java b/bdd/java/src/test/java/org/apache/iggy/bdd/ResourceOptionsSteps.java
new file mode 100644
index 000000000..c7501d926
--- /dev/null
+++ b/bdd/java/src/test/java/org/apache/iggy/bdd/ResourceOptionsSteps.java
@@ -0,0 +1,415 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iggy.bdd;
+
+import io.cucumber.java.After;
+import io.cucumber.java.en.Given;
+import io.cucumber.java.en.Then;
+import io.cucumber.java.en.When;
+import org.apache.iggy.client.blocking.tcp.IggyTcpClient;
+import org.apache.iggy.consumergroup.Consumer;
+import org.apache.iggy.identifier.StreamId;
+import org.apache.iggy.identifier.TopicId;
+import org.apache.iggy.identifier.UserId;
+import org.apache.iggy.message.HeaderKey;
+import org.apache.iggy.message.HeaderValue;
+import org.apache.iggy.message.Message;
+import org.apache.iggy.message.Partitioning;
+import org.apache.iggy.message.PollingStrategy;
+import org.apache.iggy.stream.EncryptionKeySource;
+import org.apache.iggy.stream.EncryptionKeys;
+import org.apache.iggy.stream.StreamDetails;
+import org.apache.iggy.stream.StreamOptionKey;
+import org.apache.iggy.stream.StreamOptions;
+import org.apache.iggy.stream.StreamUpdate;
+import org.apache.iggy.system.OptionSpec;
+import org.apache.iggy.system.OptionsScope;
+import org.apache.iggy.topic.TopicDetails;
+import org.apache.iggy.topic.TopicOptionKey;
+import org.apache.iggy.topic.TopicOptions;
+import org.apache.iggy.topic.TopicUpdate;
+import org.apache.iggy.user.UserOptions;
+import org.apache.iggy.user.UserStatus;
+import org.apache.iggy.user.UserUpdate;
+
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public final class ResourceOptionsSteps {
+
+ private IggyTcpClient client;
+ private Long streamId;
+ private Long topicId;
+ private Long userId;
+ private String streamName;
+ private byte[] streamKey;
+ private boolean updateRejected;
+
+ @After
+ public void close() {
+ if (client == null) {
+ return;
+ }
+ if (topicId != null && streamId != null) {
+ client.topics().deleteTopic(streamId, topicId);
+ }
+ if (streamId != null) {
+ client.streams().deleteStream(streamId);
+ }
+ if (userId != null) {
+ client.users().deleteUser(userId);
+ }
+ client.close();
+ }
+
+ @When("I create a stream named {string} with an encryption key of 32 bytes valued {int}")
+ public void createKeyedStream(String name, int value) {
+ streamKey = new byte[32];
+ Arrays.fill(streamKey, (byte) value);
+ var created = client().streams()
+ .createStream(
+ name, StreamOptions.builder().encryptionKey(streamKey).build());
+ streamId = created.id();
+ streamName = name;
+ }
+
+ @Given("a stream named {string} with an encryption key of 32 bytes valued {int} exists")
+ public void keyedStreamExists(String name, int value) {
+ createKeyedStream(name, value);
+ }
+
+ @When("I create a stream named {string} without an encryption key")
+ public void createKeylessStream(String name) {
+ var created = client().streams().createStream(name);
+ streamId = created.id();
+ streamName = name;
+ }
+
+ @Given("a stream named {string} exists")
+ public void streamExists(String name) {
+ createKeylessStream(name);
+ }
+
+ @Then("the stream should report encryption key source {string}")
+ public void streamSource(String source) {
+ assertEquals(
+ EncryptionKeySource.valueOf(source.toUpperCase()),
+ stream().encryptionKeySource().orElseThrow());
+ }
+
+ @Then("the stream encryption key fingerprint should match the key")
+ public void streamFingerprintMatches() {
+ assertNotNull(streamKey);
+ assertEquals(
+ EncryptionKeys.fingerprint(streamKey),
+ stream().encryptionKeyFingerprint().orElseThrow());
+ }
+
+ @Then("the stream should report an encryption key fingerprint")
+ public void streamFingerprintExists() {
+ assertTrue(stream().encryptionKeyFingerprint().orElseThrow().matches("[0-9a-f]{16}"));
+ }
+
+ @Then("the stream should report at least one KEK fingerprint")
+ public void kekFingerprintExists() {
+ List<String> fingerprints = stream().encryptionKekFingerprints();
+ assertFalse(fingerprints.isEmpty());
+ assertTrue(fingerprints.stream().allMatch(value -> value.matches("[0-9a-f]{16}")));
+ }
+
+ @Then("the stream should expose no encryption key material")
+ public void streamDoesNotExposeKey() {
+ var details = stream();
+ for (Map<String, HeaderValue> options : List.of(details.options(), details.derivedOptions())) {
+ for (String key : List.of("encryption_key", "encryption_key_wrapped", "critical_keys")) {
+ assertFalse(options.containsKey(key));
+ }
+ }
+ }
+
+ @When("I create a topic named {string} with {int} partition in the stream")
+ public void createTopic(String name, int partitions) {
+ var created = client().topics().createTopic(StreamId.of(requireStreamId()), name, partitions);
+ topicId = created.id();
+ }
+
+ @When("I send a message with payload {string} to the topic")
+ public void sendPayload(String payload) {
+ client().messages()
+ .sendMessages(
+ StreamId.of(requireStreamId()),
+ TopicId.of(requireTopicId()),
+ Partitioning.partitionId(0L),
+ List.of(Message.of(payload)));
+ }
+
+ @When("I send a message with payload {string} and header {string} set to {string} to the topic")
+ public void sendPayloadWithHeader(String payload, String key, String value) {
+ client().messages()
+ .sendMessages(
+ StreamId.of(requireStreamId()),
+ TopicId.of(requireTopicId()),
+ Partitioning.partitionId(0L),
+ List.of(Message.of(payload, Map.of(HeaderKey.fromString(key), HeaderValue.fromString(value)))));
+ }
+
+ @Then("polling the topic should return the payload {string}")
+ public void pollPayload(String expected) {
+ var messages = poll();
+ assertEquals(1, messages.size());
+ assertEquals(expected, new String(messages.get(0).payload(), StandardCharsets.UTF_8));
+ }
+
+ @Then("polling the topic should return the payload {string} and header {string} set to {string}")
+ public void pollPayloadWithHeader(String expected, String key, String value) {
+ var messages = poll();
+ assertEquals(1, messages.size());
+ assertEquals(expected, new String(messages.get(0).payload(), StandardCharsets.UTF_8));
+ assertEquals(
+ HeaderValue.fromString(value), messages.get(0).userHeaders().get(HeaderKey.fromString(key)));
+ }
+
+ @When("I rename the stream to {string}")
+ public void renameStream(String name) {
+ client().streams()
+ .updateStream(
+ StreamId.of(requireStreamId()),
+ StreamUpdate.builder().name(name).build());
+ streamName = name;
+ }
+
+ @Then("the stream name should be {string}")
+ public void streamName(String name) {
+ assertEquals(name, stream().name());
+ }
+
+ @When("I request a reset of the stream option {string}")
+ public void resetStreamOption(String key) {
+ try {
+ var update = StreamUpdate.builder()
+ .reset(StreamOptionKey.valueOf(key.toUpperCase()))
+ .build();
+ client().streams().updateStream(StreamId.of(requireStreamId()), update);
+ } catch (RuntimeException rejected) {
+ updateRejected = true;
+ }
+ }
+
+ @Then("the stream update should be rejected")
+ public void streamUpdateRejected() {
+ assertTrue(updateRejected);
+ }
+
+ @When("I describe the stream options")
+ public void describeStreamOptions() {
+ client().system().describeOptions(OptionsScope.Stream);
+ }
+
+ @Then("the stream option {string} should be updatable and not secret")
+ public void updatableStreamOption(String key) {
+ OptionSpec option = streamOption(key);
+ assertTrue(option.updatable());
+ assertFalse(option.secret());
+ }
+
+ @Then("the stream option {string} should be create only and secret")
+ public void secretStreamOption(String key) {
+ OptionSpec option = streamOption(key);
+ assertFalse(option.updatable());
+ assertTrue(option.secret());
+ }
+
+ @Then("the stream option {string} should be read only")
+ public void readOnlyStreamOption(String key) {
+ assertTrue(streamOption(key).readOnly());
+ }
+
+ @Given(
+ "a topic named {string} exists in the stream with message expiry {long} microseconds and max topic size {long} bytes")
+ public void topicWithOptions(String name, long expiry, long maxSize) {
+ var options = TopicOptions.builder()
+ .messageExpiry(BigInteger.valueOf(expiry))
+ .maxTopicSize(BigInteger.valueOf(maxSize))
+ .build();
+ var created = client().topics().createTopic(StreamId.of(requireStreamId()), name, 1L, options);
+ topicId = created.id();
+ }
+
+ @When("I update the topic message expiry to {long} microseconds without a name")
+ public void updateTopicExpiry(long expiry) {
+ client().topics()
+ .updateTopic(
+ StreamId.of(requireStreamId()),
+ TopicId.of(requireTopicId()),
+ TopicUpdate.builder()
+ .messageExpiry(BigInteger.valueOf(expiry))
+ .build());
+ }
+
+ @When("I reset the topic option {string}")
+ public void resetTopicOption(String key) {
+ client().topics()
+ .updateTopic(
+ StreamId.of(requireStreamId()),
+ TopicId.of(requireTopicId()),
+ TopicUpdate.builder()
+ .reset(TopicOptionKey.valueOf(key.toUpperCase()))
+ .build());
+ }
+
+ @When("I rename the topic to {string}")
+ public void renameTopic(String name) {
+ client().topics()
+ .updateTopic(
+ StreamId.of(requireStreamId()),
+ TopicId.of(requireTopicId()),
+ TopicUpdate.builder().name(name).build());
+ }
+
+ @Then("the topic name should be {string}")
+ public void topicName(String name) {
+ assertEquals(name, topic().name());
+ }
+
+ @Then("the topic message expiry should be {long} microseconds set explicitly")
+ public void topicExpiry(long expiry) {
+ assertEquals(
+ HeaderValue.fromUint64(BigInteger.valueOf(expiry)),
+ topic().options().get("message_expiry"));
+ }
+
+ @Then("the topic max topic size should be {long} bytes set explicitly")
+ public void topicMaxSize(long bytes) {
+ assertEquals(
+ HeaderValue.fromUint64(BigInteger.valueOf(bytes)),
+ topic().options().get("max_topic_size"));
+ }
+
+ @Then("the topic max topic size should be derived from the server default")
+ public void defaultTopicMaxSize() {
+ assertEquals(
+ BigInteger.ONE.shiftLeft(64).subtract(BigInteger.ONE),
+ topic().derivedOptions().get("max_topic_size").asUint64());
+ }
+
+ @Given("a user named {string} with password {string} exists")
+ public void createUser(String name, String password) {
+ var created =
+ client().users().createUser(name, password, UserStatus.Active, Optional.empty(), UserOptions.empty());
+ userId = created.id();
+ }
+
+ @When("I update the user username to {string} and status to {string}")
+ public void updateUser(String name, String status) {
+ client().users()
+ .updateUser(
+ UserId.of(requireUserId()),
+ UserUpdate.builder()
+ .username(name)
+ .status(UserStatus.valueOf(
+ status.substring(0, 1).toUpperCase() + status.substring(1)))
+ .build());
+ }
+
+ @Then("the user should have username {string}")
+ public void userName(String name) {
+ assertEquals(
+ name, client().users().getUser(requireUserId()).orElseThrow().username());
+ }
+
+ @Then("the user should have status {string}")
+ public void userStatus(String status) {
+ assertEquals(
+ status,
+ client().users()
+ .getUser(requireUserId())
+ .orElseThrow()
+ .status()
+ .name()
+ .toLowerCase());
+ }
+
+ private IggyTcpClient client() {
+ if (client == null) {
+ String[] address = TestEnvironment.serverAddress().split(":", 2);
+ client = IggyTcpClient.builder()
+ .host(address[0])
+ .port(Integer.parseInt(address[1]))
+ .build();
+ client.connect();
+ client.users().login(TestEnvironment.rootUsername(), TestEnvironment.rootPassword());
+ }
+ return client;
+ }
+
+ private StreamDetails stream() {
+ return client().streams().getStream(requireStreamId()).orElseThrow();
+ }
+
+ private TopicDetails topic() {
+ return client().topics()
+ .getTopic(StreamId.of(requireStreamId()), TopicId.of(requireTopicId()))
+ .orElseThrow();
+ }
+
+ private OptionSpec streamOption(String key) {
+ return client().system().describeOptions(OptionsScope.Stream).stream()
+ .filter(option -> option.key().equals(key))
+ .findFirst()
+ .orElseThrow();
+ }
+
+ private List<Message> poll() {
+ return client().messages()
+ .pollMessages(
+ StreamId.of(requireStreamId()),
+ TopicId.of(requireTopicId()),
+ Optional.of(0L),
+ Consumer.of(0L),
+ PollingStrategy.offset(BigInteger.ZERO),
+ 10L,
+ false)
+ .messages();
+ }
+
+ private Long requireStreamId() {
+ assertNotNull(streamId);
+ return streamId;
+ }
+
+ private Long requireTopicId() {
+ assertNotNull(topicId);
+ return topicId;
+ }
+
+ private Long requireUserId() {
+ assertNotNull(userId);
+ return userId;
+ }
+}
diff --git a/bdd/node/Dockerfile b/bdd/node/Dockerfile
index e11d1977d..a76c12b7f 100644
--- a/bdd/node/Dockerfile
+++ b/bdd/node/Dockerfile
@@ -20,6 +20,8 @@ FROM node:26-slim
COPY ./foreign/node .
COPY ./bdd/scenarios/basic_messaging.feature ./bdd/
COPY ./bdd/scenarios/raw_command.feature ./bdd/
+COPY ./bdd/scenarios/stream_options.feature ./bdd/
+COPY ./bdd/scenarios/resource_options_patch.feature ./bdd/
COPY ./bdd/scenarios/session_contract.feature ./bdd/
RUN npm ci
diff --git a/bdd/php/tests/SharedFeatureParser.php b/bdd/php/tests/SharedFeatureParser.php
index 9a0d07009..a25855645 100644
--- a/bdd/php/tests/SharedFeatureParser.php
+++ b/bdd/php/tests/SharedFeatureParser.php
@@ -20,7 +20,7 @@ declare(strict_types=1);
final class SharedFeatureParser
{
- public static function load(string $featureFile): array
+ public static function load(string $featureFile, array $excludedTags = []): array
{
if (!is_file($featureFile)) {
throw new RuntimeException("feature file not found at {$featureFile}");
@@ -35,10 +35,22 @@ final class SharedFeatureParser
$scenarios = [];
$currentScenario = null;
$currentSteps = [];
+ $currentTags = [];
+ $pendingTags = [];
$section = null;
foreach ($lines as $line) {
$line = trim($line);
- if ($line === '' || str_starts_with($line, '#') || str_starts_with($line, '@')) {
+ if ($line === '' || str_starts_with($line, '#')) {
+ continue;
+ }
+
+ if (str_starts_with($line, '@')) {
+ array_push($pendingTags, ...preg_split('/\s+/', $line));
+ continue;
+ }
+
+ if (str_starts_with($line, 'Feature:')) {
+ $pendingTags = [];
continue;
}
@@ -48,11 +60,13 @@ final class SharedFeatureParser
}
if (str_starts_with($line, 'Scenario:')) {
- if ($currentScenario !== null) {
+ if ($currentScenario !== null && array_intersect($currentTags, $excludedTags) === []) {
$scenarios[$currentScenario] = $currentSteps;
}
$currentScenario = trim(substr($line, strlen('Scenario:')));
$currentSteps = [];
+ $currentTags = $pendingTags;
+ $pendingTags = [];
$section = 'scenario';
continue;
}
@@ -74,7 +88,7 @@ final class SharedFeatureParser
}
}
- if ($currentScenario !== null) {
+ if ($currentScenario !== null && array_intersect($currentTags, $excludedTags) === []) {
$scenarios[$currentScenario] = $currentSteps;
}
if ($scenarios === []) {
diff --git a/foreign/php/src/topic.rs b/bdd/php/tests/SharedFeatureParserTest.php
similarity index 50%
copy from foreign/php/src/topic.rs
copy to bdd/php/tests/SharedFeatureParserTest.php
index d8f1772c7..be7004fac 100644
--- a/foreign/php/src/topic.rs
+++ b/bdd/php/tests/SharedFeatureParserTest.php
@@ -1,3 +1,4 @@
+<?php
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
@@ -15,40 +16,31 @@
// specific language governing permissions and limitations
// under the License.
-use ext_php_rs::{php_class, php_impl};
-use iggy::prelude::TopicDetails as RustTopicDetails;
+declare(strict_types=1);
-#[php_class]
-#[php(name = "Iggy\\TopicDetails")]
-pub struct TopicDetails {
- pub(crate) inner: RustTopicDetails,
-}
-
-impl From<RustTopicDetails> for TopicDetails {
- fn from(inner: RustTopicDetails) -> Self {
- Self { inner }
- }
-}
+require_once __DIR__ . '/SharedFeatureParser.php';
-#[php_impl]
-impl TopicDetails {
- #[php(getter)]
- pub fn id(&self) -> u32 {
- self.inner.id
- }
-
- #[php(getter)]
- pub fn name(&self) -> String {
- self.inner.name.to_string()
- }
+use PHPUnit\Framework\Attributes\Group;
+use PHPUnit\Framework\TestCase;
- #[php(getter, change_case = "snake_case")]
- pub fn messages_count(&self) -> u64 {
- self.inner.messages_count
- }
+final class SharedFeatureParserTest extends TestCase
+{
+ #[Group('stream-options')]
+ public function testGivenTaggedScenariosWhenLoadingWithExclusionsShouldKeepSupportedCases(): void
+ {
+ $cases = SharedFeatureParser::load(
+ __DIR__ . '/../../scenarios/stream_options.feature',
+ ['@describe-options', '@message-headers'],
+ );
- #[php(getter, change_case = "snake_case")]
- pub fn partitions_count(&self) -> u32 {
- self.inner.partitions_count
+ self::assertCount(4, $cases);
+ self::assertArrayNotHasKey(
+ 'Describe stream options',
+ $cases,
+ );
+ self::assertArrayNotHasKey(
+ 'Send and poll a message with headers on a keyed stream',
+ $cases,
+ );
}
}
diff --git a/bdd/php/tests/StreamOptionsFeatureTest.php b/bdd/php/tests/StreamOptionsFeatureTest.php
new file mode 100644
index 000000000..52d1e2d05
--- /dev/null
+++ b/bdd/php/tests/StreamOptionsFeatureTest.php
@@ -0,0 +1,249 @@
+<?php
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+declare(strict_types=1);
+
+require_once __DIR__ . '/SharedFeatureParser.php';
+require_once __DIR__ . '/TestEnvironment.php';
+
+use Iggy\Client as IggyClient;
+use Iggy\OptionSpec;
+use Iggy\OptionsScope;
+use Iggy\PollingStrategy;
+use Iggy\SendMessage;
+use Iggy\StreamDetails;
+use Iggy\StreamOptionKey;
+use PHPUnit\Framework\Attributes\DataProvider;
+use PHPUnit\Framework\Attributes\Group;
+use PHPUnit\Framework\TestCase;
+
+final class StreamOptionsFeatureTest extends TestCase
+{
+ private ?IggyClient $client = null;
+ private ?int $streamId = null;
+ private ?string $streamName = null;
+ private ?string $topicName = null;
+ private ?string $keyBase64 = null;
+ private bool $updateRejected = false;
+ /** @var array<string, OptionSpec> */
+ private array $streamOptionSpecs = [];
+
+ public static function scenarioCases(): array
+ {
+ return SharedFeatureParser::load(
+ __DIR__ . '/../../scenarios/stream_options.feature',
+ ['@message-headers'],
+ );
+ }
+
+ #[DataProvider('scenarioCases')]
+ #[Group('stream-options')]
+ public function testStreamOptionsScenario(string $scenarioName, array $steps): void
+ {
+ assert_true($scenarioName !== '', 'scenario name must not be empty');
+ foreach ($steps as $step) {
+ $this->runStep($step);
+ }
+ }
+
+ protected function tearDown(): void
+ {
+ if ($this->client !== null && $this->streamName !== null) {
+ cleanup_stream_with_topics(
+ $this->client,
+ $this->streamName,
+ $this->topicName === null ? [] : [$this->topicName],
+ );
+ }
+
+ parent::tearDown();
+ }
+
+ private function runStep(string $step): void
+ {
+ if ($step === 'I have a running Iggy server') {
+ $this->client = new IggyClient(TestEnvironment::serverAddress());
+ $this->client->connect();
+ $this->client->ping();
+ return;
+ }
+
+ if ($step === 'I am authenticated as the root user') {
+ $this->requireClient()->loginUser(TestEnvironment::rootUsername(), TestEnvironment::rootPassword());
+ return;
+ }
+
+ if (preg_match('/^I create a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+)$/', $step, $matches) === 1
+ || preg_match('/^a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+) exists$/', $step, $matches) === 1) {
+ $this->keyBase64 = base64_encode(str_repeat(chr((int) $matches[2]), 32));
+ $this->createStream($matches[1], $this->keyBase64);
+ return;
+ }
+
+ if (preg_match('/^I create a stream named "([^"]+)" without an encryption key$/', $step, $matches) === 1) {
+ $this->createStream($matches[1], null);
+ return;
+ }
+
+ if (preg_match('/^the stream should report encryption key source "([^"]+)"$/', $step, $matches) === 1) {
+ assert_same($matches[1], $this->requireStream()->encryption_key_source);
+ return;
+ }
+
+ if ($step === 'the stream encryption key fingerprint should match the key') {
+ assert_not_null($this->keyBase64);
+ assert_same(
+ IggyClient::encryptionKeyFingerprint($this->keyBase64),
+ $this->requireStream()->encryption_key_fingerprint,
+ );
+ return;
+ }
+
+ if ($step === 'the stream should report an encryption key fingerprint') {
+ self::assertMatchesRegularExpression('/^[0-9a-f]{16}$/', (string) $this->requireStream()->encryption_key_fingerprint);
+ return;
+ }
+
+ if ($step === 'the stream should report at least one KEK fingerprint') {
+ $fingerprints = $this->requireStream()->encryption_kek_fingerprint;
+ assert_not_null($fingerprints);
+ foreach (explode(',', $fingerprints) as $fingerprint) {
+ self::assertMatchesRegularExpression('/^[0-9a-f]{16}$/', $fingerprint);
+ }
+ return;
+ }
+
+ if ($step === 'the stream should expose no encryption key material') {
+ $stream = $this->requireStream();
+ foreach (['encryption_key', 'encryption_key_wrapped', 'critical_keys'] as $secret) {
+ self::assertArrayNotHasKey($secret, $stream->options);
+ self::assertArrayNotHasKey($secret, $stream->derived_options);
+ }
+ return;
+ }
+
+ if (preg_match('/^I create a topic named "([^"]+)" with (\d+) partition in the stream$/', $step, $matches) === 1) {
+ $this->topicName = $matches[1];
+ $this->requireClient()->createTopic($this->requireStreamId(), $this->topicName, (int) $matches[2], null, null, null, null);
+ return;
+ }
+
+ if (preg_match('/^I send a message with payload "([^"]+)" to the topic$/', $step, $matches) === 1) {
+ assert_not_null($this->topicName);
+ $this->requireClient()->sendMessages($this->requireStreamId(), $this->topicName, 0, [new SendMessage($matches[1])]);
+ return;
+ }
+
+ if (preg_match('/^polling the topic should return the payload "([^"]+)"$/', $step, $matches) === 1) {
+ assert_not_null($this->topicName);
+ $messages = $this->requireClient()->pollMessages(
+ $this->requireStreamId(), $this->topicName, 0, PollingStrategy::offset(0), 10, false,
+ );
+ assert_same([$matches[1]], collect_payloads($messages));
+ return;
+ }
+
+ if (preg_match('/^I rename the stream to "([^"]+)"$/', $step, $matches) === 1) {
+ $this->requireClient()->updateStream($this->requireStreamId(), name: $matches[1]);
+ $this->streamName = $matches[1];
+ return;
+ }
+
+ if (preg_match('/^the stream name should be "([^"]+)"$/', $step, $matches) === 1) {
+ assert_same($matches[1], $this->requireStream()->name);
+ return;
+ }
+
+ if (preg_match('/^I request a reset of the stream option "([^"]+)"$/', $step, $matches) === 1) {
+ assert_throws(
+ fn () => $this->requireClient()->updateStream(
+ $this->requireStreamId(), reset: [StreamOptionKey::from($matches[1])],
+ ),
+ 'Option cannot be updated after creation',
+ );
+ $this->updateRejected = true;
+ return;
+ }
+
+ if ($step === 'I describe the stream options') {
+ foreach ($this->requireClient()->describeOptions(OptionsScope::Stream) as $spec) {
+ $this->streamOptionSpecs[$spec->key] = $spec;
+ }
+ return;
+ }
+
+ if (preg_match('/^the stream option "([^"]+)" should be updatable and not secret$/', $step, $matches) === 1) {
+ $spec = $this->describedOption($matches[1]);
+ assert_true($spec->updatable && !$spec->secret, "{$matches[1]} must be updatable and not secret");
+ return;
+ }
+
+ if (preg_match('/^the stream option "([^"]+)" should be create only and secret$/', $step, $matches) === 1) {
+ $spec = $this->describedOption($matches[1]);
+ assert_true(!$spec->updatable && $spec->secret, "{$matches[1]} must be create only and secret");
+ return;
+ }
+
+ if (preg_match('/^the stream option "([^"]+)" should be read only$/', $step, $matches) === 1) {
+ assert_true($this->describedOption($matches[1])->read_only, "{$matches[1]} must be read only");
+ return;
+ }
+
+ if ($step === 'the stream update should be rejected') {
+ assert_true($this->updateRejected, 'the reset must be rejected');
+ return;
+ }
+
+ self::fail("Unsupported BDD step: {$step}");
+ }
+
+ private function describedOption(string $key): OptionSpec
+ {
+ assert_true(array_key_exists($key, $this->streamOptionSpecs), "{$key} must be described");
+
+ return $this->streamOptionSpecs[$key];
+ }
+
+ private function createStream(string $name, ?string $key): void
+ {
+ $this->requireClient()->createStream($name, $key);
+ $stream = $this->requireClient()->getStream($name);
+ assert_not_null($stream);
+ $this->streamId = $stream->id;
+ $this->streamName = $name;
+ }
+
+ private function requireClient(): IggyClient
+ {
+ assert_not_null($this->client);
+ return $this->client;
+ }
+
+ private function requireStreamId(): int
+ {
+ assert_not_null($this->streamId);
+ return $this->streamId;
+ }
+
+ private function requireStream(): StreamDetails
+ {
+ $stream = $this->requireClient()->getStream($this->requireStreamId());
+ assert_not_null($stream);
+ return $stream;
+ }
+}
diff --git a/bdd/python/tests/conftest.py b/bdd/python/tests/conftest.py
index 0b1ddcefa..9930e66a7 100644
--- a/bdd/python/tests/conftest.py
+++ b/bdd/python/tests/conftest.py
@@ -24,7 +24,7 @@ import os
from dataclasses import dataclass
import pytest
-from apache_iggy import IggyClient, ReceiveMessage
+from apache_iggy import IggyClient, OptionSpec, ReceiveMessage
@dataclass
@@ -42,9 +42,31 @@ class GlobalContext:
last_sent_message: str | None = None # Store message payload as string
last_raw_response: bytes | None = None
last_raw_error: RuntimeError | None = None
+ last_stream_key: bytes | None = None
+ last_update_error: RuntimeError | None = None
+ last_option_specs: list[OptionSpec] | None = None
+ last_user_id: int | None = None
last_call_error: RuntimeError | None = None
+def pytest_configure(config: pytest.Config) -> None:
+ """Register the shared feature tags as markers.
+
+ The Docker image runs pytest from /app, where bdd/python/pyproject.toml is
+ not the rootdir config, so markers declared there would never load.
+ """
+ for marker in (
+ "basic-messaging: shared basic messaging scenarios",
+ "raw-command: shared raw command scenarios",
+ "session-contract: shared session contract scenarios",
+ "stream-options: shared stream option scenarios",
+ "resource-options-patch: shared resource patch scenarios",
+ "message-headers: scenarios that assert message headers",
+ "describe-options: scenarios that assert the option catalog",
+ ):
+ config.addinivalue_line("markers", marker)
+
+
def required_env(name: str) -> str:
"""Read a variable the suite cannot run without.
diff --git a/bdd/python/tests/test_basic_messaging.py b/bdd/python/tests/test_basic_messaging.py
index b34d0cdc7..6684db06d 100644
--- a/bdd/python/tests/test_basic_messaging.py
+++ b/bdd/python/tests/test_basic_messaging.py
@@ -22,7 +22,7 @@ Basic messaging BDD test implementation for Python SDK
import asyncio
import socket
-from apache_iggy import Consumer, IggyClient, PollingStrategy, SendMessage
+from apache_iggy import Consumer, IggyClient, PollingStrategy, SendMessage, TopicOptions
from pytest_bdd import given, parsers, scenarios, then, when
# Load scenarios from the shared feature file
@@ -119,7 +119,9 @@ def create_topic(context, topic_name, stream_id, partitions):
async def _create():
await context.client.create_topic(
- stream=stream_id, name=topic_name, partitions_count=partitions
+ stream=stream_id,
+ name=topic_name,
+ options=TopicOptions(partitions_count=partitions),
)
topic = await context.client.get_topic(stream_id, topic_name)
diff --git a/bdd/python/tests/test_resource_options_patch.py b/bdd/python/tests/test_resource_options_patch.py
new file mode 100644
index 000000000..cbe3dcd7b
--- /dev/null
+++ b/bdd/python/tests/test_resource_options_patch.py
@@ -0,0 +1,248 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Resource option update BDD steps for the Python SDK.
+"""
+
+import asyncio
+import socket
+from datetime import timedelta
+from pathlib import Path
+
+from apache_iggy import (
+ IggyClient,
+ IggyExpiry,
+ MaxTopicSize,
+ OptionKey,
+ TopicDetails,
+ TopicOptions,
+ UserInfoDetails,
+ UserOptions,
+ UserStatus,
+)
+from pytest_bdd import given, parsers, scenarios, then, when
+
+FEATURE_FILE = Path("/app/features/resource_options_patch.feature")
+if not FEATURE_FILE.exists():
+ FEATURE_FILE = (
+ Path(__file__).resolve().parents[2] / "scenarios/resource_options_patch.feature"
+ )
+scenarios(str(FEATURE_FILE))
+
+
+@given("I have a running Iggy server")
+def running_server(context):
+ async def connect():
+ host, port = context.server_addr.split(":")
+ try:
+ address = f"{socket.gethostbyname(host)}:{port}"
+ except socket.gaierror:
+ address = context.server_addr
+
+ context.client = IggyClient(address)
+ await context.client.connect()
+ await context.client.ping()
+
+ asyncio.run(connect())
+
+
+@given("I am authenticated as the root user")
+def authenticated_root_user(context, root_credentials):
+ async def login():
+ await context.client.login_user(*root_credentials)
+
+ asyncio.run(login())
+
+
+@given(parsers.re(r'a stream named "(?P<name>[^"]+)" exists'))
+def stream_exists(context, name):
+ async def create():
+ stream = await context.client.create_stream(name)
+ context.last_stream_id = stream.id
+ context.last_stream_name = stream.name
+
+ asyncio.run(create())
+
+
+@given(
+ parsers.re(
+ r'a topic named "(?P<name>[^"]+)" exists in the stream with message expiry '
+ r"(?P<expiry>\d+) microseconds and max topic size (?P<max_size>\d+) bytes"
+ ),
+ converters={"expiry": int, "max_size": int},
+)
+def topic_exists_with_settings(context, name, expiry, max_size):
+ async def create():
+ await context.client.create_topic(
+ context.last_stream_id,
+ name,
+ TopicOptions(
+ message_expiry=IggyExpiry.ExpireDuration(
+ duration=timedelta(microseconds=expiry)
+ ),
+ max_topic_size=MaxTopicSize.Custom(bytes=max_size),
+ ),
+ )
+ topic = await context.client.get_topic(context.last_stream_id, name)
+ assert topic is not None
+ context.last_topic_id = topic.id
+ context.last_topic_name = topic.name
+
+ asyncio.run(create())
+
+
+@when(
+ parsers.re(
+ r"I update the topic message expiry to (?P<expiry>\d+) microseconds "
+ r"without a name"
+ ),
+ converters={"expiry": int},
+)
+def update_topic_expiry(context, expiry):
+ update_topic(
+ context,
+ message_expiry=IggyExpiry.ExpireDuration(
+ duration=timedelta(microseconds=expiry)
+ ),
+ )
+
+
+@when(parsers.re(r'I reset the topic option "(?P<key>[^"]+)"'))
+def reset_topic_option(context, key):
+ update_topic(context, reset=[option_key(key)])
+
+
+@when(parsers.re(r'I rename the topic to "(?P<name>[^"]+)"'))
+def rename_topic(context, name):
+ update_topic(context, name=name)
+
+
+@then(parsers.re(r'the topic name should be "(?P<name>[^"]+)"'))
+def topic_name_is(context, name):
+ assert fetch_topic(context).name == name
+
+
+@then(
+ parsers.re(
+ r"the topic message expiry should be (?P<expiry>\d+) microseconds "
+ r"set explicitly"
+ ),
+ converters={"expiry": int},
+)
+def topic_expiry_explicit(context, expiry):
+ topic = fetch_topic(context)
+ assert isinstance(topic.message_expiry, IggyExpiry.ExpireDuration)
+ assert topic.message_expiry.duration == timedelta(microseconds=expiry)
+ assert topic.options.to_scalar_dict()["message_expiry"] == expiry
+
+
+@then(
+ parsers.re(
+ r"the topic max topic size should be (?P<max_size>\d+) bytes set explicitly"
+ ),
+ converters={"max_size": int},
+)
+def topic_max_size_explicit(context, max_size):
+ topic = fetch_topic(context)
+ assert isinstance(topic.max_topic_size, MaxTopicSize.Custom)
+ assert topic.max_topic_size.bytes == max_size
+ assert topic.options.to_scalar_dict()["max_topic_size"] == max_size
+
+
+@then("the topic max topic size should be derived from the server default")
+def topic_max_size_derived(context):
+ topic = fetch_topic(context)
+ assert "max_topic_size" not in topic.options.to_scalar_dict()
+ assert "max_topic_size" in topic.derived_options.to_scalar_dict()
+
+
+@given(
+ parsers.re(
+ r'a user named "(?P<username>[^"]+)" with password "(?P<password>[^"]+)" exists'
+ )
+)
+def user_exists(context, username, password):
+ async def create():
+ user = await context.client.create_user(
+ username, password, options=UserOptions()
+ )
+ context.last_user_id = user.id
+
+ asyncio.run(create())
+
+
+@when(
+ parsers.re(
+ r'I update the user username to "(?P<username>[^"]+)" and status to '
+ r'"(?P<status>[^"]+)"'
+ )
+)
+def update_user(context, username, status):
+ async def update():
+ await context.client.update_user(
+ context.last_user_id, username=username, status=user_status(status)
+ )
+
+ asyncio.run(update())
+
+
+@then(parsers.re(r'the user should have username "(?P<username>[^"]+)"'))
+def user_has_username(context, username):
+ assert fetch_user(context).username == username
+
+
+@then(parsers.re(r'the user should have status "(?P<status>[^"]+)"'))
+def user_has_status(context, status):
+ assert fetch_user(context).status == user_status(status)
+
+
+def update_topic(context, **changes) -> None:
+ async def update():
+ await context.client.update_topic(
+ context.last_stream_id, context.last_topic_id, **changes
+ )
+
+ asyncio.run(update())
+
+
+def fetch_topic(context) -> TopicDetails:
+ async def fetch():
+ return await context.client.get_topic(
+ context.last_stream_id, context.last_topic_id
+ )
+
+ topic = asyncio.run(fetch())
+ assert topic is not None
+ return topic
+
+
+def fetch_user(context) -> UserInfoDetails:
+ async def fetch():
+ return await context.client.get_user(context.last_user_id)
+
+ user = asyncio.run(fetch())
+ assert user is not None
+ return user
+
+
+def option_key(key: str) -> OptionKey:
+ return getattr(OptionKey, "".join(part.title() for part in key.split("_")))
+
+
+def user_status(status: str) -> UserStatus:
+ return getattr(UserStatus, status.title())
diff --git a/bdd/python/tests/test_stream_options.py b/bdd/python/tests/test_stream_options.py
new file mode 100644
index 000000000..2a4b9e005
--- /dev/null
+++ b/bdd/python/tests/test_stream_options.py
@@ -0,0 +1,331 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Stream options BDD steps for the Python SDK.
+"""
+
+import asyncio
+import base64
+import socket
+from pathlib import Path
+
+from apache_iggy import (
+ Consumer,
+ IggyClient,
+ IggyExpiry,
+ OptionKey,
+ PollingStrategy,
+ SendMessage,
+ StreamDetails,
+ StreamOptions,
+ TopicOptions,
+ encryption_key_fingerprint,
+)
+from pytest_bdd import given, parsers, scenarios, then, when
+
+FEATURE_FILE = Path("/app/features/stream_options.feature")
+if not FEATURE_FILE.exists():
+ FEATURE_FILE = (
+ Path(__file__).resolve().parents[2] / "scenarios/stream_options.feature"
+ )
+scenarios(str(FEATURE_FILE))
+
+KEY_LENGTH = 32
+FINGERPRINT_LENGTH = 16
+PARTITION_ID = 0
+NEVER_RETURNED_KEYS = {"encryption_key", "encryption_key_wrapped", "critical_keys"}
+
+
+@given("I have a running Iggy server")
+def running_server(context):
+ async def connect():
+ host, port = context.server_addr.split(":")
+ try:
+ address = f"{socket.gethostbyname(host)}:{port}"
+ except socket.gaierror:
+ address = context.server_addr
+
+ context.client = IggyClient(address)
+ await context.client.connect()
+ await context.client.ping()
+
+ asyncio.run(connect())
+
+
+@given("I am authenticated as the root user")
+def authenticated_root_user(context, root_credentials):
+ async def login():
+ await context.client.login_user(*root_credentials)
+
+ asyncio.run(login())
+
+
+@when(
+ parsers.re(
+ r'I create a stream named "(?P<name>[^"]+)" with an encryption key '
+ r"of 32 bytes valued (?P<byte>\d+)"
+ ),
+ converters={"byte": int},
+)
+def create_stream_with_key(context, name, byte):
+ create_keyed_stream(context, name, byte)
+
+
+@given(
+ parsers.re(
+ r'a stream named "(?P<name>[^"]+)" with an encryption key '
+ r"of 32 bytes valued (?P<byte>\d+) exists"
+ ),
+ converters={"byte": int},
+)
+def keyed_stream_exists(context, name, byte):
+ create_keyed_stream(context, name, byte)
+
+
+@when(
+ parsers.re(r'I create a stream named "(?P<name>[^"]+)" without an encryption key')
+)
+def create_stream_without_key(context, name):
+ async def create():
+ stream = await context.client.create_stream(name)
+ context.last_stream_id = stream.id
+ context.last_stream_name = stream.name
+ context.last_stream_key = None
+
+ asyncio.run(create())
+
+
+@then(parsers.re(r'the stream should report encryption key source "(?P<source>[^"]+)"'))
+def stream_reports_source(context, source):
+ assert fetch_stream(context).encryption_key_source == source
+
+
+@then("the stream encryption key fingerprint should match the key")
+def stream_fingerprint_matches_key(context):
+ assert context.last_stream_key is not None
+ expected = encryption_key_fingerprint(context.last_stream_key)
+ assert fetch_stream(context).encryption_key_fingerprint == expected
+
+
+@then("the stream should report an encryption key fingerprint")
+def stream_reports_fingerprint(context):
+ fingerprint = fetch_stream(context).encryption_key_fingerprint
+ assert fingerprint is not None
+ assert len(fingerprint) == FINGERPRINT_LENGTH
+
+
+@then("the stream should report at least one KEK fingerprint")
+def stream_reports_kek_fingerprint(context):
+ assert fetch_stream(context).encryption_kek_fingerprints
+
+
+@then("the stream should expose no encryption key material")
+def stream_exposes_no_key_material(context):
+ stream = fetch_stream(context)
+ explicit = stream.options.to_scalar_dict()
+ derived = stream.derived_options.to_scalar_dict()
+ assert not NEVER_RETURNED_KEYS & (set(explicit) | set(derived))
+ if context.last_stream_key is not None:
+ values = [*explicit.values(), *derived.values()]
+ assert context.last_stream_key not in values
+ assert base64.b64encode(context.last_stream_key).decode() not in values
+
+
+@when(
+ parsers.re(
+ r'I create a topic named "(?P<name>[^"]+)" with (?P<partitions>\d+) '
+ r"partitions? in the stream"
+ ),
+ converters={"partitions": int},
+)
+def create_topic_in_stream(context, name, partitions):
+ async def create():
+ await context.client.create_topic(
+ context.last_stream_id,
+ name,
+ TopicOptions(
+ partitions_count=partitions,
+ message_expiry=IggyExpiry.NeverExpire(),
+ ),
+ )
+ topic = await context.client.get_topic(context.last_stream_id, name)
+ assert topic is not None
+ context.last_topic_id = topic.id
+ context.last_topic_name = topic.name
+ context.last_topic_partitions = partitions
+
+ asyncio.run(create())
+
+
+@when(parsers.re(r'I send a message with payload "(?P<payload>[^"]+)" to the topic'))
+def send_payload(context, payload):
+ send_message(context, SendMessage(payload))
+
+
+@when(
+ parsers.re(
+ r'I send a message with payload "(?P<payload>[^"]+)" and header '
+ r'"(?P<header>[^"]+)" set to "(?P<value>[^"]+)" to the topic'
+ )
+)
+def send_payload_with_header(context, payload, header, value):
+ send_message(context, SendMessage(payload, user_headers={header: value}))
+
+
+@then(parsers.re(r'polling the topic should return the payload "(?P<payload>[^"]+)"'))
+def poll_returns_payload(context, payload):
+ message = poll_first(context)
+ assert message.payload() == payload.encode()
+
+
+@then(
+ parsers.re(
+ r'polling the topic should return the payload "(?P<payload>[^"]+)" '
+ r'and header "(?P<header>[^"]+)" set to "(?P<value>[^"]+)"'
+ )
+)
+def poll_returns_payload_and_header(context, payload, header, value):
+ message = poll_first(context)
+ assert message.payload() == payload.encode()
+ headers = message.user_headers()
+ assert headers is not None
+ assert headers.to_scalar_dict()[header] == value
+
+
+@when(parsers.re(r'I rename the stream to "(?P<name>[^"]+)"'))
+def rename_stream(context, name):
+ async def rename():
+ await context.client.update_stream(context.last_stream_id, name=name)
+
+ asyncio.run(rename())
+
+
+@then(parsers.re(r'the stream name should be "(?P<name>[^"]+)"'))
+def stream_name_is(context, name):
+ assert fetch_stream(context).name == name
+
+
+@when(parsers.re(r'I request a reset of the stream option "(?P<key>[^"]+)"'))
+def request_stream_reset(context, key):
+ async def reset():
+ try:
+ await context.client.update_stream(
+ context.last_stream_id, reset=[option_key(key)]
+ )
+ context.last_update_error = None
+ except RuntimeError as error:
+ context.last_update_error = error
+
+ asyncio.run(reset())
+
+
+@then("the stream update should be rejected")
+def stream_update_rejected(context):
+ assert context.last_update_error is not None
+
+
+@when("I describe the stream options")
+def describe_stream_options(context):
+ async def describe():
+ context.last_option_specs = await context.client.describe_options("stream")
+
+ asyncio.run(describe())
+
+
+@then(
+ parsers.re(r'the stream option "(?P<key>[^"]+)" should be updatable and not secret')
+)
+def option_updatable(context, key):
+ spec = option_spec(context, key)
+ assert spec.updatable and not spec.secret and not spec.read_only
+
+
+@then(
+ parsers.re(r'the stream option "(?P<key>[^"]+)" should be create only and secret')
+)
+def option_create_only_secret(context, key):
+ spec = option_spec(context, key)
+ assert not spec.updatable and not spec.resettable
+ assert spec.secret and not spec.read_only
+
+
+@then(parsers.re(r'the stream option "(?P<key>[^"]+)" should be read only'))
+def option_read_only(context, key):
+ spec = option_spec(context, key)
+ assert spec.read_only and not spec.updatable and not spec.secret
+
+
+def create_keyed_stream(context, name: str, byte: int) -> None:
+ key = bytes([byte]) * KEY_LENGTH
+
+ async def create():
+ stream = await context.client.create_stream(
+ name, StreamOptions(encryption_key=key)
+ )
+ context.last_stream_id = stream.id
+ context.last_stream_name = stream.name
+ context.last_stream_key = key
+
+ asyncio.run(create())
+
+
+def fetch_stream(context) -> StreamDetails:
+ async def fetch():
+ return await context.client.get_stream(context.last_stream_id)
+
+ stream = asyncio.run(fetch())
+ assert stream is not None
+ return stream
+
+
+def send_message(context, message: SendMessage) -> None:
+ async def send():
+ await context.client.send_messages(
+ stream=context.last_stream_id,
+ topic=context.last_topic_id,
+ partitioning=PARTITION_ID,
+ messages=[message],
+ )
+
+ asyncio.run(send())
+
+
+def poll_first(context):
+ async def poll():
+ return await context.client.poll_messages(
+ stream=context.last_stream_id,
+ topic=context.last_topic_id,
+ consumer=Consumer.Single(1),
+ partition_id=PARTITION_ID,
+ polling_strategy=PollingStrategy.Offset(value=0),
+ count=1,
+ auto_commit=False,
+ )
+
+ messages = asyncio.run(poll())
+ assert messages
+ return messages[0]
+
+
+def option_key(key: str) -> OptionKey:
+ return getattr(OptionKey, "".join(part.title() for part in key.split("_")))
+
+
+def option_spec(context, key: str):
+ assert context.last_option_specs is not None
+ return next(spec for spec in context.last_option_specs if spec.key == key)
diff --git a/bdd/python/uv.lock b/bdd/python/uv.lock
index cb82f871f..4a602d52b 100644
--- a/bdd/python/uv.lock
+++ b/bdd/python/uv.lock
@@ -8,7 +8,7 @@ exclude-newer-span = "P7D"
[[package]]
name = "apache-iggy"
-version = "0.9.1.dev1"
+version = "0.9.1.dev2"
source = { directory = "../../foreign/python" }
[package.metadata]
diff --git a/bdd/rust/Cargo.toml b/bdd/rust/Cargo.toml
index 03207ef69..cb7249d56 100644
--- a/bdd/rust/Cargo.toml
+++ b/bdd/rust/Cargo.toml
@@ -60,5 +60,15 @@ name = "stream_crud"
harness = false
required-features = ["bdd"]
+[[test]]
+name = "stream_options"
+harness = false
+required-features = ["bdd"]
+
+[[test]]
+name = "resource_options_patch"
+harness = false
+required-features = ["bdd"]
+
[lints]
workspace = true
diff --git a/bdd/rust/tests/common/global_context.rs b/bdd/rust/tests/common/global_context.rs
index 3313c003e..b7d36b86f 100644
--- a/bdd/rust/tests/common/global_context.rs
+++ b/bdd/rust/tests/common/global_context.rs
@@ -18,7 +18,7 @@
use bytes::Bytes;
use cucumber::World;
use iggy::clients::client::IggyClient;
-use iggy::prelude::{IggyError, IggyMessage, PolledMessages};
+use iggy::prelude::{IggyError, IggyMessage, OptionSpec, PolledMessages, StreamEncryptionKey};
#[derive(Debug, World, Default)]
pub struct GlobalContext {
@@ -33,5 +33,9 @@ pub struct GlobalContext {
pub last_polled_messages: Option<PolledMessages>,
pub last_sent_message: Option<IggyMessage>,
pub last_raw_result: Option<Result<Bytes, IggyError>>,
+ pub last_stream_key: Option<StreamEncryptionKey>,
+ pub last_update_error: Option<IggyError>,
+ pub last_option_specs: Vec<OptionSpec>,
+ pub last_user_id: Option<u32>,
pub last_call_result: Option<Result<(), IggyError>>,
}
diff --git a/foreign/csharp/Iggy_SDK/Contracts/Http/Auth/UpdateUserRequest.cs b/bdd/rust/tests/resource_options_patch.rs
similarity index 71%
copy from foreign/csharp/Iggy_SDK/Contracts/Http/Auth/UpdateUserRequest.cs
copy to bdd/rust/tests/resource_options_patch.rs
index 7ef9c59f6..43739ea60 100644
--- a/foreign/csharp/Iggy_SDK/Contracts/Http/Auth/UpdateUserRequest.cs
+++ b/bdd/rust/tests/resource_options_patch.rs
@@ -15,8 +15,17 @@
// specific language governing permissions and limitations
// under the License.
-using Apache.Iggy.Enums;
+pub(crate) mod common;
+pub(crate) mod helpers;
+pub(crate) mod steps;
-namespace Apache.Iggy.Contracts.Http.Auth;
+use crate::common::global_context::GlobalContext;
+use cucumber::World;
-internal sealed record UpdateUserRequest(string? Username, UserStatus? UserStatus);
+#[tokio::main]
+async fn main() {
+ GlobalContext::cucumber()
+ .fail_on_skipped()
+ .run_and_exit("../../bdd/scenarios/resource_options_patch.feature")
+ .await;
+}
diff --git a/bdd/rust/tests/steps/leader_redirection.rs b/bdd/rust/tests/steps/leader_redirection.rs
index dde838f91..bcbf891f6 100644
--- a/bdd/rust/tests/steps/leader_redirection.rs
+++ b/bdd/rust/tests/steps/leader_redirection.rs
@@ -180,7 +180,7 @@ async fn when_create_stream(world: &mut LeaderContext, stream_name: String) {
.expect("Client should be available");
let stream = client
- .create_stream(&stream_name)
+ .create_stream(&stream_name, &StreamCreateOptions::default())
.await
.expect("Should be able to create stream");
diff --git a/bdd/rust/tests/steps/mod.rs b/bdd/rust/tests/steps/mod.rs
index e6c88141d..a091ca2cc 100644
--- a/bdd/rust/tests/steps/mod.rs
+++ b/bdd/rust/tests/steps/mod.rs
@@ -19,6 +19,7 @@ pub mod auth;
pub mod leader_redirection;
pub mod messages;
pub mod raw_command;
+pub mod resource_options;
pub mod server;
pub mod session;
pub mod streams;
diff --git a/bdd/rust/tests/steps/resource_options.rs b/bdd/rust/tests/steps/resource_options.rs
new file mode 100644
index 000000000..de6d6dd78
--- /dev/null
+++ b/bdd/rust/tests/steps/resource_options.rs
@@ -0,0 +1,509 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::common::global_context::GlobalContext;
+use cucumber::{given, then, when};
+use iggy::prelude::{
+ Consumer, HeaderKey, HeaderValue, Identifier, IggyExpiry, IggyMessage, MaxTopicSize,
+ MessageClient, OptionKey, OptionSpec, OptionsScope, Partitioning, PollingStrategy,
+ StreamClient, StreamCreateOptions, StreamDetails, StreamEncryptionKey, StreamEncryptionSource,
+ StreamUpdateOptions, SystemClient, TopicClient, TopicCreateOptions, TopicDetails,
+ TopicUpdateOptions, UserClient, UserCreateOptions, UserStatus, UserUpdateOptions,
+};
+use std::collections::{BTreeMap, BTreeSet};
+use std::str::FromStr;
+
+const KEY_LENGTH: usize = 32;
+const PARTITION_ID: u32 = 0;
+
+#[when(
+ regex = r#"^I create a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+)$"#
+)]
+pub async fn when_create_stream_with_key(world: &mut GlobalContext, name: String, byte: u8) {
+ create_keyed_stream(world, &name, byte).await;
+}
+
+#[given(
+ regex = r#"^a stream named "([^"]+)" with an encryption key of 32 bytes valued (\d+) exists$"#
+)]
+pub async fn given_keyed_stream(world: &mut GlobalContext, name: String, byte: u8) {
+ create_keyed_stream(world, &name, byte).await;
+}
+
+#[when(regex = r#"^I create a stream named "([^"]+)" without an encryption key$"#)]
+pub async fn when_create_stream_without_key(world: &mut GlobalContext, name: String) {
+ let client = world.client.as_ref().expect("client");
+ let stream = client
+ .create_stream(&name, &StreamCreateOptions::default())
+ .await
+ .expect("create stream");
+ world.last_stream_id = Some(stream.id);
+ world.last_stream_name = Some(stream.name);
+ world.last_stream_key = None;
+}
+
+#[given(regex = r#"^a stream named "([^"]+)" exists$"#)]
+pub async fn given_plain_stream(world: &mut GlobalContext, name: String) {
+ when_create_stream_without_key(world, name).await;
+}
+
+#[then(regex = r#"^the stream should report encryption key source "([^"]+)"$"#)]
+pub async fn then_stream_source(world: &mut GlobalContext, source: String) {
+ let stream = fetch_stream(world).await;
+ let expected = StreamEncryptionSource::from_str(&source).expect("known source");
+ assert_eq!(stream.encryption_key_source(), Some(expected));
+}
+
+#[then("the stream encryption key fingerprint should match the key")]
+pub async fn then_fingerprint_matches_key(world: &mut GlobalContext) {
+ let stream = fetch_stream(world).await;
+ let key = world.last_stream_key.as_ref().expect("stream key");
+ assert_eq!(stream.encryption_key_fingerprint(), Some(key.fingerprint()));
+}
+
+#[then("the stream should report an encryption key fingerprint")]
+pub async fn then_stream_reports_fingerprint(world: &mut GlobalContext) {
+ let stream = fetch_stream(world).await;
+ let fingerprint = stream.encryption_key_fingerprint().expect("fingerprint");
+ assert_eq!(fingerprint.len(), 16);
+}
+
+#[then("the stream should report at least one KEK fingerprint")]
+pub async fn then_stream_reports_kek(world: &mut GlobalContext) {
+ let stream = fetch_stream(world).await;
+ assert!(!stream.encryption_kek_fingerprints().is_empty());
+}
+
+#[then("the stream should expose no encryption key material")]
+pub async fn then_no_key_material(world: &mut GlobalContext) {
+ let stream = fetch_stream(world).await;
+ for secret in [OptionKey::EncryptionKey, OptionKey::EncryptionKeyWrapped] {
+ assert!(
+ !stream.options.contains_key(&HeaderKey::from(secret)),
+ "{secret} must never be returned"
+ );
+ }
+}
+
+#[when(regex = r#"^I create a topic named "([^"]+)" with (\d+) partitions? in the stream$"#)]
+pub async fn when_create_topic_in_stream(
+ world: &mut GlobalContext,
+ name: String,
+ partitions_count: u32,
+) {
+ let client = world.client.as_ref().expect("client");
+ let topic = client
+ .create_topic(
+ &stream_identifier(world),
+ &name,
+ &TopicCreateOptions {
+ partitions_count: Some(partitions_count),
+ message_expiry: Some(IggyExpiry::NeverExpire),
+ ..TopicCreateOptions::default()
+ },
+ )
+ .await
+ .expect("create topic");
+ world.last_topic_id = Some(topic.id);
+ world.last_topic_name = Some(topic.name);
+}
+
+#[when(regex = r#"^I send a message with payload "([^"]+)" to the topic$"#)]
+pub async fn when_send_payload(world: &mut GlobalContext, payload: String) {
+ send_message(world, &payload, BTreeMap::new()).await;
+}
+
+#[when(
+ regex = r#"^I send a message with payload "([^"]+)" and header "([^"]+)" set to "([^"]+)" to the topic$"#
+)]
+pub async fn when_send_payload_with_header(
+ world: &mut GlobalContext,
+ payload: String,
+ header: String,
+ value: String,
+) {
+ let headers = BTreeMap::from([(
+ HeaderKey::from_str(&header).expect("header key"),
+ HeaderValue::from_str(&value).expect("header value"),
+ )]);
+ send_message(world, &payload, headers).await;
+}
+
+#[then(regex = r#"^polling the topic should return the payload "([^"]+)"$"#)]
+pub async fn then_poll_returns_payload(world: &mut GlobalContext, payload: String) {
+ let message = poll_first(world).await;
+ assert_eq!(message.payload.as_ref(), payload.as_bytes());
+}
+
+#[then(
+ regex = r#"^polling the topic should return the payload "([^"]+)" and header "([^"]+)" set to "([^"]+)"$"#
+)]
+pub async fn then_poll_returns_payload_and_header(
+ world: &mut GlobalContext,
+ payload: String,
+ header: String,
+ value: String,
+) {
+ let message = poll_first(world).await;
+ assert_eq!(message.payload.as_ref(), payload.as_bytes());
+ let headers = message
+ .user_headers_map()
+ .expect("readable headers")
+ .expect("headers present");
+ assert_eq!(
+ headers.get(&HeaderKey::from_str(&header).expect("header key")),
+ Some(&HeaderValue::from_str(&value).expect("header value"))
+ );
+}
+
+#[when(regex = r#"^I rename the stream to "([^"]+)"$"#)]
+pub async fn when_rename_stream(world: &mut GlobalContext, name: String) {
+ let client = world.client.as_ref().expect("client");
+ client
+ .update_stream(
+ &stream_identifier(world),
+ &StreamUpdateOptions {
+ name: Some(name),
+ ..StreamUpdateOptions::default()
+ },
+ )
+ .await
+ .expect("rename stream");
+}
+
+#[then(regex = r#"^the stream name should be "([^"]+)"$"#)]
+pub async fn then_stream_name(world: &mut GlobalContext, name: String) {
+ assert_eq!(fetch_stream(world).await.name, name);
+}
+
+#[when(regex = r#"^I request a reset of the stream option "([^"]+)"$"#)]
+pub async fn when_reset_stream_option(world: &mut GlobalContext, key: String) {
+ let client = world.client.as_ref().expect("client");
+ let result = client
+ .update_stream(
+ &stream_identifier(world),
+ &StreamUpdateOptions {
+ reset: BTreeSet::from([OptionKey::from_str(&key).expect("known key")]),
+ ..StreamUpdateOptions::default()
+ },
+ )
+ .await;
+ world.last_update_error = result.err();
+}
+
+#[then("the stream update should be rejected")]
+pub async fn then_stream_update_rejected(world: &mut GlobalContext) {
+ assert!(world.last_update_error.is_some(), "the update must fail");
+}
+
+#[when("I describe the stream options")]
+pub async fn when_describe_stream_options(world: &mut GlobalContext) {
+ let client = world.client.as_ref().expect("client");
+ world.last_option_specs = client
+ .describe_options(OptionsScope::Stream)
+ .await
+ .expect("describe options");
+}
+
+#[then(regex = r#"^the stream option "([^"]+)" should be updatable and not secret$"#)]
+pub async fn then_option_updatable(world: &mut GlobalContext, key: String) {
+ let spec = option_spec(world, &key);
+ assert!(spec.updatable && !spec.secret && !spec.read_only);
+}
+
+#[then(regex = r#"^the stream option "([^"]+)" should be create only and secret$"#)]
+pub async fn then_option_create_only_secret(world: &mut GlobalContext, key: String) {
+ let spec = option_spec(world, &key);
+ assert!(!spec.updatable && !spec.resettable && spec.secret && !spec.read_only);
+}
+
+#[then(regex = r#"^the stream option "([^"]+)" should be read only$"#)]
+pub async fn then_option_read_only(world: &mut GlobalContext, key: String) {
+ let spec = option_spec(world, &key);
+ assert!(spec.read_only && !spec.updatable && !spec.secret);
+}
+
+#[given(
+ regex = r#"^a topic named "([^"]+)" exists in the stream with message expiry (\d+) microseconds and max topic size (\d+) bytes$"#
+)]
+pub async fn given_topic_with_settings(
+ world: &mut GlobalContext,
+ name: String,
+ expiry: u64,
+ max_size: u64,
+) {
+ let client = world.client.as_ref().expect("client");
+ let topic = client
+ .create_topic(
+ &stream_identifier(world),
+ &name,
+ &TopicCreateOptions {
+ partitions_count: Some(1),
+ message_expiry: Some(IggyExpiry::from(expiry)),
+ max_topic_size: Some(MaxTopicSize::from(max_size)),
+ ..TopicCreateOptions::default()
+ },
+ )
+ .await
+ .expect("create topic");
+ world.last_topic_id = Some(topic.id);
+ world.last_topic_name = Some(topic.name);
+}
+
+#[when(regex = r"^I update the topic message expiry to (\d+) microseconds without a name$")]
+pub async fn when_update_topic_expiry(world: &mut GlobalContext, expiry: u64) {
+ update_topic(
+ world,
+ TopicUpdateOptions {
+ message_expiry: Some(IggyExpiry::from(expiry)),
+ ..TopicUpdateOptions::default()
+ },
+ )
+ .await;
+}
+
+#[when(regex = r#"^I reset the topic option "([^"]+)"$"#)]
+pub async fn when_reset_topic_option(world: &mut GlobalContext, key: String) {
+ update_topic(
+ world,
+ TopicUpdateOptions {
+ reset: BTreeSet::from([OptionKey::from_str(&key).expect("known key")]),
+ ..TopicUpdateOptions::default()
+ },
+ )
+ .await;
+}
+
+#[when(regex = r#"^I rename the topic to "([^"]+)"$"#)]
+pub async fn when_rename_topic(world: &mut GlobalContext, name: String) {
+ update_topic(
+ world,
+ TopicUpdateOptions {
+ name: Some(name),
+ ..TopicUpdateOptions::default()
+ },
+ )
+ .await;
+}
+
+#[then(regex = r#"^the topic name should be "([^"]+)"$"#)]
+pub async fn then_topic_name(world: &mut GlobalContext, name: String) {
+ assert_eq!(fetch_topic(world).await.name, name);
+}
+
+#[then(regex = r"^the topic message expiry should be (\d+) microseconds set explicitly$")]
+pub async fn then_topic_expiry_explicit(world: &mut GlobalContext, expiry: u64) {
+ let topic = fetch_topic(world).await;
+ assert_eq!(
+ topic.resolved_options().message_expiry,
+ IggyExpiry::from(expiry)
+ );
+ assert!(topic_option_explicit(&topic, OptionKey::MessageExpiry));
+}
+
+#[then(regex = r"^the topic max topic size should be (\d+) bytes set explicitly$")]
+pub async fn then_topic_max_size_explicit(world: &mut GlobalContext, max_size: u64) {
+ let topic = fetch_topic(world).await;
+ assert_eq!(
+ topic.resolved_options().max_topic_size,
+ MaxTopicSize::from(max_size)
+ );
+ assert!(topic_option_explicit(&topic, OptionKey::MaxTopicSize));
+}
+
+#[then("the topic max topic size should be derived from the server default")]
+pub async fn then_topic_max_size_derived(world: &mut GlobalContext) {
+ let topic = fetch_topic(world).await;
+ assert!(!topic_option_explicit(&topic, OptionKey::MaxTopicSize));
+}
+
+#[given(regex = r#"^a user named "([^"]+)" with password "([^"]+)" exists$"#)]
+pub async fn given_user(world: &mut GlobalContext, username: String, password: String) {
+ let client = world.client.as_ref().expect("client");
+ let user = client
+ .create_user(
+ &username,
+ &password,
+ UserStatus::Active,
+ None,
+ &UserCreateOptions::default(),
+ )
+ .await
+ .expect("create user");
+ world.last_user_id = Some(user.id);
+}
+
+#[when(regex = r#"^I update the user username to "([^"]+)" and status to "([^"]+)"$"#)]
+pub async fn when_update_user(world: &mut GlobalContext, username: String, status: String) {
+ let client = world.client.as_ref().expect("client");
+ client
+ .update_user(
+ &user_identifier(world),
+ &UserUpdateOptions {
+ username: Some(username),
+ status: Some(UserStatus::from_str(&status).expect("known status")),
+ ..UserUpdateOptions::default()
+ },
+ )
+ .await
+ .expect("update user");
+}
+
+#[then(regex = r#"^the user should have username "([^"]+)"$"#)]
+pub async fn then_user_username(world: &mut GlobalContext, username: String) {
+ let client = world.client.as_ref().expect("client");
+ let user = client
+ .get_user(&user_identifier(world))
+ .await
+ .expect("get user")
+ .expect("user exists");
+ assert_eq!(user.username, username);
+}
+
+#[then(regex = r#"^the user should have status "([^"]+)"$"#)]
+pub async fn then_user_status(world: &mut GlobalContext, status: String) {
+ let client = world.client.as_ref().expect("client");
+ let user = client
+ .get_user(&user_identifier(world))
+ .await
+ .expect("get user")
+ .expect("user exists");
+ assert_eq!(
+ user.status,
+ UserStatus::from_str(&status).expect("known status")
+ );
+}
+
+async fn create_keyed_stream(world: &mut GlobalContext, name: &str, byte: u8) {
+ let key = StreamEncryptionKey::try_from([byte; KEY_LENGTH].as_slice()).expect("32 bytes");
+ let client = world.client.as_ref().expect("client");
+ let stream = client
+ .create_stream(
+ name,
+ &StreamCreateOptions {
+ encryption_key: Some(key.clone()),
+ ..StreamCreateOptions::default()
+ },
+ )
+ .await
+ .expect("create keyed stream");
+ world.last_stream_id = Some(stream.id);
+ world.last_stream_name = Some(stream.name);
+ world.last_stream_key = Some(key);
+}
+
+async fn fetch_stream(world: &GlobalContext) -> StreamDetails {
+ let client = world.client.as_ref().expect("client");
+ client
+ .get_stream(&stream_identifier(world))
+ .await
+ .expect("get stream")
+ .expect("stream exists")
+}
+
+async fn fetch_topic(world: &GlobalContext) -> TopicDetails {
+ let client = world.client.as_ref().expect("client");
+ client
+ .get_topic(&stream_identifier(world), &topic_identifier(world))
+ .await
+ .expect("get topic")
+ .expect("topic exists")
+}
+
+async fn update_topic(world: &GlobalContext, options: TopicUpdateOptions) {
+ let client = world.client.as_ref().expect("client");
+ client
+ .update_topic(
+ &stream_identifier(world),
+ &topic_identifier(world),
+ &options,
+ )
+ .await
+ .expect("update topic");
+}
+
+async fn send_message(
+ world: &GlobalContext,
+ payload: &str,
+ headers: BTreeMap<HeaderKey, HeaderValue>,
+) {
+ let client = world.client.as_ref().expect("client");
+ let builder = IggyMessage::builder().payload(payload.to_owned().into());
+ let message = if headers.is_empty() {
+ builder.build()
+ } else {
+ builder.user_headers(headers).build()
+ }
+ .expect("message");
+ client
+ .send_messages(
+ &stream_identifier(world),
+ &topic_identifier(world),
+ &Partitioning::partition_id(PARTITION_ID),
+ &mut [message],
+ )
+ .await
+ .expect("send message");
+}
+
+async fn poll_first(world: &GlobalContext) -> IggyMessage {
+ let client = world.client.as_ref().expect("client");
+ let polled = client
+ .poll_messages(
+ &stream_identifier(world),
+ &topic_identifier(world),
+ Some(PARTITION_ID),
+ &Consumer::default(),
+ &PollingStrategy::offset(0),
+ 1,
+ false,
+ )
+ .await
+ .expect("poll messages");
+ polled
+ .messages
+ .into_iter()
+ .next()
+ .expect("one message polled")
+}
+
+fn option_spec<'a>(world: &'a GlobalContext, key: &str) -> &'a OptionSpec {
+ world
+ .last_option_specs
+ .iter()
+ .find(|spec| spec.key == key)
+ .expect("described key")
+}
+
+fn topic_option_explicit(topic: &TopicDetails, key: OptionKey) -> bool {
+ topic
+ .options
+ .get(&HeaderKey::from(key))
+ .is_some_and(|option| option.explicit)
+}
+
+fn stream_identifier(world: &GlobalContext) -> Identifier {
+ Identifier::numeric(world.last_stream_id.expect("stream created")).expect("stream id")
+}
+
+fn topic_identifier(world: &GlobalContext) -> Identifier {
+ Identifier::numeric(world.last_topic_id.expect("topic created")).expect("topic id")
+}
+
+fn user_identifier(world: &GlobalContext) -> Identifier {
+ Identifier::numeric(world.last_user_id.expect("user created")).expect("user id")
+}
diff --git a/bdd/rust/tests/steps/streams.rs b/bdd/rust/tests/steps/streams.rs
index f94ef4027..9d9ce3e2a 100644
--- a/bdd/rust/tests/steps/streams.rs
+++ b/bdd/rust/tests/steps/streams.rs
@@ -17,6 +17,7 @@
use crate::common::global_context::GlobalContext;
use cucumber::{given, then, when};
+use iggy::prelude::StreamCreateOptions;
use iggy::prelude::{Identifier, StreamClient, StreamUpdateOptions};
#[given("I have no streams in the system")]
@@ -96,8 +97,10 @@ pub async fn when_update_stream_name(world: &mut GlobalContext, stream_name: Str
client
.update_stream(
&Identifier::numeric(stream_id).expect("Stream ID should be valid"),
- &stream_name,
- &StreamUpdateOptions::default(),
+ &StreamUpdateOptions {
+ name: Some(stream_name.to_owned()),
+ ..StreamUpdateOptions::default()
+ },
)
.await
.expect("Should be able to update stream");
@@ -142,7 +145,7 @@ pub async fn then_get_stream_returns_no_stream(world: &mut GlobalContext) {
async fn create_stream(world: &mut GlobalContext, stream_name: &str) {
let client = world.client.as_ref().expect("Client should be available");
let stream = client
- .create_stream(stream_name)
+ .create_stream(stream_name, &StreamCreateOptions::default())
.await
.expect("Should be able to create stream");
diff --git a/foreign/csharp/Iggy_SDK/Contracts/Http/Auth/UpdateUserRequest.cs b/bdd/rust/tests/stream_options.rs
similarity index 71%
copy from foreign/csharp/Iggy_SDK/Contracts/Http/Auth/UpdateUserRequest.cs
copy to bdd/rust/tests/stream_options.rs
index 7ef9c59f6..ce5c56385 100644
--- a/foreign/csharp/Iggy_SDK/Contracts/Http/Auth/UpdateUserRequest.cs
+++ b/bdd/rust/tests/stream_options.rs
@@ -15,8 +15,17 @@
// specific language governing permissions and limitations
// under the License.
-using Apache.Iggy.Enums;
+pub(crate) mod common;
+pub(crate) mod helpers;
+pub(crate) mod steps;
-namespace Apache.Iggy.Contracts.Http.Auth;
+use crate::common::global_context::GlobalContext;
+use cucumber::World;
-internal sealed record UpdateUserRequest(string? Username, UserStatus? UserStatus);
+#[tokio::main]
+async fn main() {
+ GlobalContext::cucumber()
+ .fail_on_skipped()
+ .run_and_exit("../../bdd/scenarios/stream_options.feature")
+ .await;
+}
diff --git a/bdd/scenarios/resource_options_patch.feature b/bdd/scenarios/resource_options_patch.feature
new file mode 100644
index 000000000..348156f67
--- /dev/null
+++ b/bdd/scenarios/resource_options_patch.feature
@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+@resource-options-patch
+Feature: Resource option updates
+ As a developer using Apache Iggy
+ I want to change only the settings I name on a topic or a user
+ So that an update never resends or silently resets the others
+
+ Background:
+ Given I have a running Iggy server
+ And I am authenticated as the root user
+
+ Scenario: Update one topic setting
+ Given a stream named "patch-topic-stream" exists
+ And a topic named "patch-topic" exists in the stream with message expiry 3600000000 microseconds and max topic size 1073741824 bytes
+ When I update the topic message expiry to 7200000000 microseconds without a name
+ Then the topic name should be "patch-topic"
+ And the topic message expiry should be 7200000000 microseconds set explicitly
+ And the topic max topic size should be 1073741824 bytes set explicitly
+
+ Scenario: Reset a topic setting to the server default
+ Given a stream named "patch-reset-stream" exists
+ And a topic named "patch-reset" exists in the stream with message expiry 3600000000 microseconds and max topic size 1073741824 bytes
+ When I reset the topic option "max_topic_size"
+ Then the topic max topic size should be derived from the server default
+ And the topic message expiry should be 3600000000 microseconds set explicitly
+
+ Scenario: Rename a topic
+ Given a stream named "patch-rename-stream" exists
+ And a topic named "patch-rename" exists in the stream with message expiry 3600000000 microseconds and max topic size 1073741824 bytes
+ When I rename the topic to "patch-renamed"
+ Then the topic name should be "patch-renamed"
+ And the topic message expiry should be 3600000000 microseconds set explicitly
+ And the topic max topic size should be 1073741824 bytes set explicitly
+
+ Scenario: Update a user username and status
+ Given a user named "patch-user" with password "patch-password" exists
+ When I update the user username to "patch-user-renamed" and status to "inactive"
+ Then the user should have username "patch-user-renamed"
+ And the user should have status "inactive"
diff --git a/bdd/scenarios/stream_options.feature b/bdd/scenarios/stream_options.feature
new file mode 100644
index 000000000..c1a56e72c
--- /dev/null
+++ b/bdd/scenarios/stream_options.feature
@@ -0,0 +1,74 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+@stream-options
+Feature: Stream options
+ As a developer using Apache Iggy
+ I want to give a stream its own encryption key through typed options
+ So that each stream keeps its data at rest under its own key
+
+ Background:
+ Given I have a running Iggy server
+ And I am authenticated as the root user
+
+ Scenario: Create a stream with its own key
+ When I create a stream named "stream-options-own-key" with an encryption key of 32 bytes valued 7
+ Then the stream should report encryption key source "stream"
+ And the stream encryption key fingerprint should match the key
+ And the stream should report at least one KEK fingerprint
+ And the stream should expose no encryption key material
+
+ Scenario: Create a stream with a server generated key
+ When I create a stream named "stream-options-generated" without an encryption key
+ Then the stream should report encryption key source "server"
+ And the stream should report an encryption key fingerprint
+ And the stream should report at least one KEK fingerprint
+ And the stream should expose no encryption key material
+ When I create a topic named "events" with 1 partition in the stream
+ And I send a message with payload "generated-key-payload" to the topic
+ Then polling the topic should return the payload "generated-key-payload"
+
+ @message-headers
+ Scenario: Send and poll a message with headers on a keyed stream
+ Given a stream named "stream-options-custom" with an encryption key of 32 bytes valued 9 exists
+ When I create a topic named "events" with 1 partition in the stream
+ And I send a message with payload "custom-key-payload" and header "trace" set to "abc" to the topic
+ Then polling the topic should return the payload "custom-key-payload" and header "trace" set to "abc"
+
+ Scenario: Rename a keyed stream
+ Given a stream named "stream-options-rename" with an encryption key of 32 bytes valued 11 exists
+ When I create a topic named "events" with 1 partition in the stream
+ And I send a message with payload "before-rename" to the topic
+ And I rename the stream to "stream-options-renamed"
+ Then the stream name should be "stream-options-renamed"
+ And the stream should report encryption key source "stream"
+ And the stream encryption key fingerprint should match the key
+ And polling the topic should return the payload "before-rename"
+
+ Scenario: Stream key reset is rejected
+ Given a stream named "stream-options-reset" with an encryption key of 32 bytes valued 13 exists
+ When I request a reset of the stream option "encryption_key"
+ Then the stream update should be rejected
+ And the stream name should be "stream-options-reset"
+ And the stream encryption key fingerprint should match the key
+
+ @describe-options
+ Scenario: Describe stream options
+ When I describe the stream options
+ Then the stream option "name" should be updatable and not secret
+ And the stream option "encryption_key" should be create only and secret
+ And the stream option "encryption_key_fingerprint" should be read only
diff --git a/core/ai/mcp/Cargo.toml b/core/ai/mcp/Cargo.toml
index a912c0255..1fc411cbe 100644
--- a/core/ai/mcp/Cargo.toml
+++ b/core/ai/mcp/Cargo.toml
@@ -17,7 +17,7 @@
[package]
name = "iggy-mcp"
-version = "0.5.1-edge.1"
+version = "0.5.1-edge.2"
description = "MCP Server for Iggy message streaming platform"
edition = "2024"
license = "Apache-2.0"
diff --git a/core/ai/mcp/src/service/mod.rs b/core/ai/mcp/src/service/mod.rs
index f12952258..5fbe23481 100644
--- a/core/ai/mcp/src/service/mod.rs
+++ b/core/ai/mcp/src/service/mod.rs
@@ -18,10 +18,11 @@
use iggy::prelude::{
Client, ClusterClient, CompressionAlgorithm, Consumer, ConsumerGroupClient,
ConsumerOffsetClient, Identifier, IggyClient, IggyError, IggyExpiry, IggyMessage,
- IggyTimestamp, MaxTopicSize, MessageClient, PartitionClient, Partitioning,
+ IggyTimestamp, MaxTopicSize, MessageClient, OptionKey, PartitionClient, Partitioning,
PersonalAccessTokenClient, PollingKind, PollingStrategy, SegmentClient, StreamClient,
- StreamUpdateOptions, SystemClient, SystemSnapshotType, TopicClient, TopicCreateOptions,
- TopicUpdateOptions, UserClient, UserStatus, UserUpdateOptions,
+ StreamCreateOptions, StreamUpdateOptions, SystemClient, SystemSnapshotType, TopicClient,
+ TopicCreateOptions, TopicUpdateOptions, UserClient, UserCreateOptions, UserStatus,
+ UserUpdateOptions,
};
use requests::*;
use rmcp::{
@@ -31,6 +32,9 @@ use rmcp::{
tool, tool_handler, tool_router,
};
use serde::Serialize;
+use std::collections::BTreeSet;
+use std::fmt::Display;
+use std::str::FromStr;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tracing::error;
@@ -90,18 +94,23 @@ impl IggyService {
Parameters(CreateStream { name }): Parameters<CreateStream>,
) -> Result<CallToolResult, ErrorData> {
self.permissions.ensure_create()?;
- request(self.client.create_stream(&name).await)
+ request(
+ self.client
+ .create_stream(&name, &StreamCreateOptions::default())
+ .await,
+ )
}
#[tool(description = "Update stream")]
pub async fn update_stream(
&self,
- Parameters(UpdateStream { stream_id, name }): Parameters<UpdateStream>,
+ Parameters(update): Parameters<UpdateStream>,
) -> Result<CallToolResult, ErrorData> {
self.permissions.ensure_update()?;
+ let stream_id = id(&update.stream_id)?;
request(
self.client
- .update_stream(&id(&stream_id)?, &name, &StreamUpdateOptions::default())
+ .update_stream(&stream_id, &StreamUpdateOptions::from(update))
.await,
)
}
@@ -189,28 +198,15 @@ impl IggyService {
#[tool(description = "Update topic")]
pub async fn update_topic(
&self,
- Parameters(UpdateTopic {
- stream_id,
- topic_id,
- name,
- compression_algorithm,
- message_expiry,
- max_size,
- options,
- }): Parameters<UpdateTopic>,
+ Parameters(update): Parameters<UpdateTopic>,
) -> Result<CallToolResult, ErrorData> {
self.permissions.ensure_update()?;
- // Absent means "leave alone" now, so an unparsable value must not
- // silently become a reset to the server default.
- let update_options = TopicUpdateOptions {
- compression_algorithm: compression_algorithm.and_then(|ca| ca.parse().ok()),
- message_expiry: message_expiry.and_then(|me| me.parse().ok()),
- max_topic_size: max_size.and_then(|ms| ms.parse().ok()),
- raw: options,
- };
+ let stream_id = id(&update.stream_id)?;
+ let topic_id = id(&update.topic_id)?;
+ let update_options = TopicUpdateOptions::try_from(update)?;
request(
self.client
- .update_topic(&id(&stream_id)?, &id(&topic_id)?, &name, &update_options)
+ .update_topic(&stream_id, &topic_id, &update_options)
.await,
)
}
@@ -637,7 +633,8 @@ impl IggyService {
) -> Result<CallToolResult, ErrorData> {
self.permissions.ensure_create()?;
let expiry = expiry
- .and_then(|expiry| expiry.parse().ok())
+ .map(|expiry| parsed(&expiry, "personal access token expiry"))
+ .transpose()?
.unwrap_or_default();
request(
self.client
@@ -689,7 +686,13 @@ impl IggyService {
let permissions = permissions.map(|p| p.into());
request(
self.client
- .create_user(&username, &password, status, permissions)
+ .create_user(
+ &username,
+ &password,
+ status,
+ permissions,
+ &UserCreateOptions::default(),
+ )
.await,
)
}
@@ -697,28 +700,13 @@ impl IggyService {
#[tool(description = "Update user")]
pub async fn update_user(
&self,
- Parameters(UpdateUser {
- user_id,
- username,
- active,
- }): Parameters<UpdateUser>,
+ Parameters(update): Parameters<UpdateUser>,
) -> Result<CallToolResult, ErrorData> {
self.permissions.ensure_update()?;
- let status = active.map(|active| {
- if active {
- UserStatus::Active
- } else {
- UserStatus::Inactive
- }
- });
+ let user_id = id(&update.user_id)?;
request(
self.client
- .update_user(
- &id(&user_id)?,
- username.as_deref(),
- status,
- &UserUpdateOptions::default(),
- )
+ .update_user(&user_id, &UserUpdateOptions::from(update))
.await,
)
}
@@ -775,6 +763,88 @@ impl ServerHandler for IggyService {
}
}
+impl From<UpdateStream> for StreamUpdateOptions {
+ fn from(update: UpdateStream) -> Self {
+ let (reset, raw_reset) = split_reset(update.reset);
+ StreamUpdateOptions {
+ name: update.name,
+ reset,
+ raw_reset,
+ ..StreamUpdateOptions::default()
+ }
+ }
+}
+
+impl TryFrom<UpdateTopic> for TopicUpdateOptions {
+ type Error = ErrorData;
+
+ fn try_from(update: UpdateTopic) -> Result<Self, Self::Error> {
+ let (reset, raw_reset) = split_reset(update.reset);
+ Ok(TopicUpdateOptions {
+ name: update.name,
+ message_expiry: update
+ .message_expiry
+ .map(|value| parsed(&value, "message expiry"))
+ .transpose()?,
+ max_topic_size: update
+ .max_size
+ .map(|value| parsed(&value, "maximum size"))
+ .transpose()?,
+ reset,
+ raw_reset,
+ raw: update.options,
+ })
+ }
+}
+
+impl From<UpdateUser> for UserUpdateOptions {
+ fn from(update: UpdateUser) -> Self {
+ let (reset, raw_reset) = split_reset(update.reset);
+ UserUpdateOptions {
+ username: update.username,
+ status: update.active.map(|active| {
+ if active {
+ UserStatus::Active
+ } else {
+ UserStatus::Inactive
+ }
+ }),
+ reset,
+ raw_reset,
+ raw: update.options,
+ }
+ }
+}
+
+/// Split reset names into keys this build checks before sending and newer keys
+/// the server decides on.
+fn split_reset(names: Vec<String>) -> (BTreeSet<OptionKey>, BTreeSet<String>) {
+ let mut known = BTreeSet::new();
+ let mut newer = BTreeSet::new();
+ for name in names {
+ match name.parse::<OptionKey>() {
+ Ok(key) => {
+ known.insert(key);
+ }
+ Err(_) => {
+ newer.insert(name);
+ }
+ }
+ }
+ (known, newer)
+}
+
+fn parsed<T: FromStr>(value: &str, what: &str) -> Result<T, ErrorData>
+where
+ T::Err: Display,
+{
+ value.parse().map_err(|error: T::Err| {
+ let message = format!("Failed to parse {what}. {error}");
+ error!(message);
+ ErrorData::invalid_request(message, None)
+ })
+}
+
fn id(id: &str) -> Result<Identifier, ErrorData> {
Identifier::from_str_value(id).map_err(|e| {
let message = format!("Failed to parse identifier. {e}");
@@ -814,6 +884,7 @@ fn operation_id_value(value: &str) -> Result<u64, ErrorData> {
#[cfg(test)]
mod tests {
use super::*;
+ use std::collections::BTreeMap;
#[test]
fn given_uncertain_or_blocked_writes_when_reporting_errors_should_preserve_exact_operation_ids()
@@ -828,4 +899,82 @@ mod tests {
assert_eq!(operation_id_value(operation).unwrap(), u64::MAX);
}
}
+
+ #[test]
+ fn given_update_topic_values_when_converting_should_parse_them_into_typed_options() {
+ let update = update_topic(
+ r#"{"name":"events-v2","message_expiry":"7days","max_size":"15GB","options":{"future_key":"1"}}"#,
+ );
+
+ let options = TopicUpdateOptions::try_from(update).expect("valid values convert");
+
+ assert_eq!(
+ options,
+ TopicUpdateOptions {
+ name: Some("events-v2".to_owned()),
+ message_expiry: Some("7days".parse().expect("valid expiry")),
+ max_topic_size: Some("15GB".parse().expect("valid size")),
+ raw: BTreeMap::from([("future_key".to_owned(), "1".to_owned())]),
+ ..TopicUpdateOptions::default()
+ }
+ );
+ }
+
+ #[test]
+ fn given_unparsable_update_topic_value_when_converting_should_refuse() {
+ for body in [r#"{"message_expiry":"soon"}"#, r#"{"max_size":"big"}"#] {
+ assert!(
+ TopicUpdateOptions::try_from(update_topic(body)).is_err(),
+ "{body}"
+ );
+ }
+ }
+
+ #[test]
+ fn given_reset_names_when_converting_should_send_known_typed_and_newer_raw() {
+ let update = update_topic(r#"{"reset":["message_expiry","future_key"]}"#);
+
+ let options = TopicUpdateOptions::try_from(update).expect("reset names convert");
+
+ assert_eq!(
+ options,
+ TopicUpdateOptions {
+ reset: BTreeSet::from([OptionKey::MessageExpiry]),
+ raw_reset: BTreeSet::from(["future_key".to_owned()]),
+ ..TopicUpdateOptions::default()
+ }
+ );
+ }
+
+ #[test]
+ fn given_stream_and_user_resets_when_converting_should_keep_the_name_optional() {
+ let stream: UpdateStream =
+ serde_json::from_str(r#"{"stream_id":"orders","reset":["future_key"]}"#)
+ .expect("a reset without a name parses");
+ let user: UpdateUser = serde_json::from_str(
+ r#"{"user_id":"alice","reset":["future_key"],"options":{"newer_key":"x"}}"#,
+ )
+ .expect("a user patch parses");
+
+ assert_eq!(
+ StreamUpdateOptions::from(stream),
+ StreamUpdateOptions {
+ raw_reset: BTreeSet::from(["future_key".to_owned()]),
+ ..StreamUpdateOptions::default()
+ }
+ );
+ assert_eq!(
+ UserUpdateOptions::from(user),
+ UserUpdateOptions {
+ raw_reset: BTreeSet::from(["future_key".to_owned()]),
+ raw: BTreeMap::from([("newer_key".to_owned(), "x".to_owned())]),
+ ..UserUpdateOptions::default()
+ }
+ );
+ }
+
+ fn update_topic(fields: &str) -> UpdateTopic {
+ let ids = r#"{"stream_id":"orders","topic_id":"events","#;
+ serde_json::from_str(&format!("{ids}{}", &fields[1..])).expect("valid update_topic body")
+ }
}
diff --git a/core/ai/mcp/src/service/requests.rs b/core/ai/mcp/src/service/requests.rs
index dd3dbccb0..05eb380d0 100644
--- a/core/ai/mcp/src/service/requests.rs
+++ b/core/ai/mcp/src/service/requests.rs
@@ -36,17 +36,24 @@ pub struct GetStream {
}
#[derive(Debug, Deserialize, JsonSchema)]
+#[serde(deny_unknown_fields)]
pub struct CreateStream {
#[schemars(description = "stream name (required, must be unique)")]
pub name: String,
}
#[derive(Debug, Deserialize, JsonSchema)]
+#[serde(deny_unknown_fields)]
pub struct UpdateStream {
#[schemars(description = "stream identifier (name or number)")]
pub stream_id: String,
- #[schemars(description = "stream name (required, must be unique)")]
- pub name: String,
+ #[schemars(description = "new name, must be unique (optional)")]
+ pub name: Option<String>,
+ #[schemars(
+ description = "setting names to return to their defaults, checked by the Iggy server (optional)"
+ )]
+ #[serde(default)]
+ pub reset: Vec<String>,
}
#[derive(Debug, Deserialize, JsonSchema)]
@@ -99,6 +106,7 @@ pub struct CreateTopic {
}
#[derive(Debug, Deserialize, JsonSchema)]
+#[serde(deny_unknown_fields)]
pub struct UpdateTopic {
#[schemars(description = "stream identifier (name or number)")]
pub stream_id: String,
@@ -106,22 +114,25 @@ pub struct UpdateTopic {
#[schemars(description = "topic identifier (name or number)")]
pub topic_id: String,
- #[schemars(description = "name (required, must be unique)")]
- pub name: String,
+ #[schemars(description = "new name, must be unique in the stream (optional)")]
+ pub name: Option<String>,
- #[schemars(description = "compression algorithm (optional, can be one of 'none', 'gzip')")]
- pub compression_algorithm: Option<String>,
-
- #[schemars(description = "message expiry (optional)")]
+ #[schemars(description = "message expiry like '7 days' or 'unlimited' (optional)")]
pub message_expiry: Option<String>,
- #[schemars(description = "maximum size (optional)")]
+ #[schemars(description = "maximum size like '15 GB' or 'unlimited' (optional)")]
pub max_size: Option<String>,
#[schemars(
- description = "options to change, as string key-values; only compression_algorithm, \
- message_expiry and max_topic_size may be updated, and a key left out \
- keeps its current value (optional)"
+ description = "setting names to return to their defaults, e.g. message_expiry. Keys are \
+ validated by the Iggy server (optional)"
+ )]
+ #[serde(default)]
+ pub reset: Vec<String>,
+
+ #[schemars(
+ description = "other updatable settings as string key-values, a key left out keeps its \
+ current value (optional)"
)]
#[serde(default)]
pub options: BTreeMap<String, String>,
@@ -377,6 +388,19 @@ pub struct UpdateUser {
#[schemars(description = "active (optional)")]
pub active: Option<bool>,
+
+ #[schemars(
+ description = "setting names to return to their defaults, checked by the Iggy server (optional)"
+ )]
+ #[serde(default)]
+ pub reset: Vec<String>,
+
+ #[schemars(
+ description = "other updatable settings as string key-values, a key left out keeps its \
+ current value (optional)"
+ )]
+ #[serde(default)]
+ pub options: BTreeMap<String, String>,
}
#[derive(Debug, Deserialize, JsonSchema)]
@@ -550,3 +574,47 @@ impl From<TopicPermissions> for prelude::TopicPermissions {
}
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use rmcp::schemars::schema_for;
+
+ #[test]
+ fn given_stream_tool_schema_when_describing_should_exclude_encryption_key() {
+ for schema in [schema_for!(CreateStream), schema_for!(UpdateStream)] {
+ let properties = schema
+ .get("properties")
+ .and_then(|properties| properties.as_object())
+ .expect("object schema");
+ for secret_route in ["encryption_key", "options", "raw"] {
+ assert!(
+ !properties.contains_key(secret_route),
+ "stream tools must not accept {secret_route}"
+ );
+ }
+ }
+
+ let smuggled = [
+ r#"{"name":"orders","encryption_key":"AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8="}"#,
+ r#"{"name":"orders","options":{"encryption_key":"AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8="}}"#,
+ ];
+ for body in smuggled {
+ assert!(
+ serde_json::from_str::<CreateStream>(body).is_err(),
+ "{body}"
+ );
+ }
+ let rename: UpdateStream =
+ serde_json::from_str(r#"{"stream_id":"orders","name":"orders-v2"}"#)
+ .expect("a rename still parses");
+ assert_eq!(rename.name.as_deref(), Some("orders-v2"));
+ }
+
+ #[test]
+ fn given_update_topic_with_retired_compression_when_parsing_should_refuse() {
+ let body = r#"{"stream_id":"orders","topic_id":"events","name":"events-v2","compression_algorithm":"gzip"}"#;
+
+ assert!(serde_json::from_str::<UpdateTopic>(body).is_err());
+ }
+}
diff --git a/core/bench/Cargo.toml b/core/bench/Cargo.toml
index 0b9079867..f05e136d7 100644
--- a/core/bench/Cargo.toml
+++ b/core/bench/Cargo.toml
@@ -17,7 +17,7 @@
[package]
name = "iggy-bench"
-version = "0.6.1-edge.1"
+version = "0.6.1-edge.2"
edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/apache/iggy"
diff --git a/core/bench/src/benchmarks/benchmark.rs b/core/bench/src/benchmarks/benchmark.rs
index c5359de94..a37a2194c 100644
--- a/core/bench/src/benchmarks/benchmark.rs
+++ b/core/bench/src/benchmarks/benchmark.rs
@@ -116,7 +116,9 @@ pub trait Benchmarkable: Send {
}
info!("Creating the test stream '{}'", stream_name);
- client.create_stream(&stream_name).await?;
+ client
+ .create_stream(&stream_name, &StreamCreateOptions::default())
+ .await?;
let topic_name = "topic-1".to_string();
let max_topic_size = self
.args()
diff --git a/core/binary_protocol/Cargo.toml b/core/binary_protocol/Cargo.toml
index 9c5e86db8..531d4784d 100644
--- a/core/binary_protocol/Cargo.toml
+++ b/core/binary_protocol/Cargo.toml
@@ -17,7 +17,7 @@
[package]
name = "iggy_binary_protocol"
-version = "0.11.1-edge.1"
+version = "0.11.1-edge.2"
description = "Wire protocol types and codec for the Iggy binary protocol. Shared between server and SDK."
edition = "2024"
rust-version.workspace = true
@@ -33,6 +33,7 @@ readme = "README.md"
bytemuck = { workspace = true }
bytes = { workspace = true }
enumset = { workspace = true }
+ring = { workspace = true }
secrecy = { workspace = true }
serde = { workspace = true }
thiserror = { workspace = true }
diff --git a/core/binary_protocol/src/codes.rs b/core/binary_protocol/src/codes.rs
index d90f4463d..ab8e8f9f4 100644
--- a/core/binary_protocol/src/codes.rs
+++ b/core/binary_protocol/src/codes.rs
@@ -28,6 +28,7 @@ pub const GET_CLUSTER_METADATA_CODE: u32 = 12;
pub const DESCRIBE_OPTIONS_CODE: u32 = 13;
// Code 14 is retired.
pub const BIND_SESSION_CODE: u32 = 15;
+pub const UPDATE_RESOURCE_CODE: u32 = 16;
pub const GET_ME_CODE: u32 = 20;
pub const GET_CLIENT_CODE: u32 = 21;
pub const GET_CLIENTS_CODE: u32 = 22;
@@ -37,7 +38,6 @@ pub const GET_USER_CODE: u32 = 31;
pub const GET_USERS_CODE: u32 = 32;
pub const CREATE_USER_CODE: u32 = 33;
pub const DELETE_USER_CODE: u32 = 34;
-pub const UPDATE_USER_CODE: u32 = 35;
pub const UPDATE_PERMISSIONS_CODE: u32 = 36;
pub const CHANGE_PASSWORD_CODE: u32 = 37;
pub const LOGIN_USER_CODE: u32 = 38;
@@ -73,7 +73,6 @@ pub const GET_STREAM_CODE: u32 = 200;
pub const GET_STREAMS_CODE: u32 = 201;
pub const CREATE_STREAM_CODE: u32 = 202;
pub const DELETE_STREAM_CODE: u32 = 203;
-pub const UPDATE_STREAM_CODE: u32 = 204;
// 205 was PURGE_STREAM. Never reuse it.
// -- Topics --
@@ -81,7 +80,6 @@ pub const GET_TOPIC_CODE: u32 = 300;
pub const GET_TOPICS_CODE: u32 = 301;
pub const CREATE_TOPIC_CODE: u32 = 302;
pub const DELETE_TOPIC_CODE: u32 = 303;
-pub const UPDATE_TOPIC_CODE: u32 = 304;
// 305 was PURGE_TOPIC. Never reuse it.
// -- Partitions --
@@ -115,7 +113,7 @@ pub const fn command_name(code: u32) -> Result<&'static str, WireError> {
mod tests {
use super::*;
- const RETIRED_CODES: &[u32] = &[14, 102, 103, 104, 123, 124, 205, 305];
+ const RETIRED_CODES: &[u32] = &[14, 35, 102, 103, 104, 123, 124, 204, 205, 304, 305];
const ALL_CODES: &[u32] = &[
PING_CODE,
@@ -124,6 +122,7 @@ mod tests {
GET_CLUSTER_METADATA_CODE,
DESCRIBE_OPTIONS_CODE,
BIND_SESSION_CODE,
+ UPDATE_RESOURCE_CODE,
GET_ME_CODE,
GET_CLIENT_CODE,
GET_CLIENTS_CODE,
@@ -131,7 +130,6 @@ mod tests {
GET_USERS_CODE,
CREATE_USER_CODE,
DELETE_USER_CODE,
- UPDATE_USER_CODE,
UPDATE_PERMISSIONS_CODE,
CHANGE_PASSWORD_CODE,
LOGIN_USER_CODE,
@@ -152,12 +150,10 @@ mod tests {
GET_STREAMS_CODE,
CREATE_STREAM_CODE,
DELETE_STREAM_CODE,
- UPDATE_STREAM_CODE,
GET_TOPIC_CODE,
GET_TOPICS_CODE,
CREATE_TOPIC_CODE,
DELETE_TOPIC_CODE,
- UPDATE_TOPIC_CODE,
CREATE_PARTITIONS_CODE,
DELETE_PARTITIONS_CODE,
DELETE_SEGMENTS_CODE,
diff --git a/core/binary_protocol/src/consensus/header.rs b/core/binary_protocol/src/consensus/header.rs
index ee095693b..0d64b42c1 100644
--- a/core/binary_protocol/src/consensus/header.rs
+++ b/core/binary_protocol/src/consensus/header.rs
@@ -298,13 +298,10 @@ pub struct RequestHeader {
pub reserved_frame: [u8; 66],
pub client: u128,
- /// Optional XXH3 integrity stamp over the client's request payload. Zero
- /// means unstamped; the server verifies a nonzero stamp before routing or
- /// rewriting the body. The Rust SDK stamps metadata/session operations
- /// and `DeleteSegments`; partition and non-replicated operations leave it
- /// zero. Retry identity uses the client, session, request, group and
- /// operation, so matching retries replay the first receipt without
- /// comparing stamps.
+ /// Random identifier for a logical metadata or session request. The client
+ /// keeps it across retries and reconnect resends. Partition and
+ /// non-replicated operations leave it zero. The server seals the prepared
+ /// body separately with `checksum_body`.
pub request_checksum: u128,
pub timestamp: u64,
pub request: u64,
diff --git a/core/binary_protocol/src/consensus/operation.rs b/core/binary_protocol/src/consensus/operation.rs
index 7961e2167..44f99dafb 100644
--- a/core/binary_protocol/src/consensus/operation.rs
+++ b/core/binary_protocol/src/consensus/operation.rs
@@ -64,14 +64,26 @@ pub enum Operation {
TruncatePartition = 68,
FinalizeSession = 70,
CompleteLifecycle = 71,
+ /// Prepare-resolved `UpdateResource` on a stream: column changes extracted,
+ /// resets resolved to their defaults. Claimed by the streams state machine.
+ UpdateStreamResolved = 72,
+ /// Prepare-resolved `UpdateResource` on a topic.
+ UpdateTopicResolved = 73,
+ /// Prepare-resolved `UpdateResource` on a user. Claimed by the users state
+ /// machine.
+ UpdateUserResolved = 74,
+ /// Server-originated: replace encrypted streams' wrap sets during a
+ /// key-encryption key rotation. Proposed by the metadata primary, no
+ /// client wire code.
+ RewrapStreamKeys = 75,
// Metadata operations (shard 0)
CreateStream = 128,
- UpdateStream = 129,
+ // 129 was UpdateStream. Never reuse it.
DeleteStream = 130,
// 131 was PurgeStream. Never reuse it.
CreateTopic = 132,
- UpdateTopic = 133,
+ // 133 was UpdateTopic. Never reuse it.
DeleteTopic = 134,
// 135 was PurgeTopic. Never reuse it.
CreatePartitions = 136,
@@ -84,7 +96,7 @@ pub enum Operation {
CreateConsumerGroup = 139,
DeleteConsumerGroup = 140,
CreateUser = 141,
- UpdateUser = 142,
+ // 142 was UpdateUser. Never reuse it.
DeleteUser = 143,
ChangePassword = 144,
UpdatePermissions = 145,
@@ -92,6 +104,9 @@ pub enum Operation {
DeletePersonalAccessToken = 147,
JoinConsumerGroup = 148,
LeaveConsumerGroup = 149,
+ /// Generic option patch for a stream, topic or user. Prepare rewrites it to
+ /// the resolved operation of the target's scope.
+ UpdateResource = 150,
// Partition operations (routed by namespace)
SendMessages = 160,
@@ -128,6 +143,9 @@ impl Operation {
match self {
Self::CreateTopicWithAssignments => Self::CreateTopic,
Self::CreatePartitionsWithAssignments => Self::CreatePartitions,
+ Self::UpdateStreamResolved | Self::UpdateTopicResolved | Self::UpdateUserResolved => {
+ Self::UpdateResource
+ }
Self::TruncatePartition => Self::DeleteSegments,
operation => *operation,
}
@@ -172,17 +190,14 @@ impl Operation {
matches!(
self,
Self::CreateStream
- | Self::UpdateStream
| Self::DeleteStream
| Self::CreateTopic
- | Self::UpdateTopic
| Self::DeleteTopic
| Self::CreatePartitions
| Self::DeletePartitions
| Self::CreateConsumerGroup
| Self::DeleteConsumerGroup
| Self::CreateUser
- | Self::UpdateUser
| Self::DeleteUser
| Self::ChangePassword
| Self::UpdatePermissions
@@ -190,6 +205,7 @@ impl Operation {
| Self::DeletePersonalAccessToken
| Self::JoinConsumerGroup
| Self::LeaveConsumerGroup
+ | Self::UpdateResource
)
}
@@ -269,6 +285,10 @@ impl Operation {
| Self::NonReplicated
| Self::CreateTopicWithAssignments
| Self::CreatePartitionsWithAssignments
+ | Self::UpdateStreamResolved
+ | Self::UpdateTopicResolved
+ | Self::UpdateUserResolved
+ | Self::RewrapStreamKeys
| Self::RemoveConsumerGroupMember
| Self::CompleteConsumerGroupRevocation
| Self::TruncatePartition
@@ -279,10 +299,8 @@ impl Operation {
| Self::TransitionPartitionHistory
| Self::RetireConsumerGroupOwners => None,
Self::CreateStream
- | Self::UpdateStream
| Self::DeleteStream
| Self::CreateTopic
- | Self::UpdateTopic
| Self::DeleteTopic
| Self::CreatePartitions
| Self::DeletePartitions
@@ -290,7 +308,6 @@ impl Operation {
| Self::CreateConsumerGroup
| Self::DeleteConsumerGroup
| Self::CreateUser
- | Self::UpdateUser
| Self::DeleteUser
| Self::ChangePassword
| Self::UpdatePermissions
@@ -298,6 +315,7 @@ impl Operation {
| Self::DeletePersonalAccessToken
| Self::JoinConsumerGroup
| Self::LeaveConsumerGroup
+ | Self::UpdateResource
| Self::SendMessages
| Self::StoreConsumerOffset
| Self::DeleteConsumerOffset => match crate::dispatch::lookup_by_operation(*self) {
@@ -327,10 +345,8 @@ mod tests {
fn command_code_roundtrip() {
let ops = [
Operation::CreateStream,
- Operation::UpdateStream,
Operation::DeleteStream,
Operation::CreateTopic,
- Operation::UpdateTopic,
Operation::DeleteTopic,
Operation::CreatePartitions,
Operation::DeletePartitions,
@@ -338,7 +354,6 @@ mod tests {
Operation::CreateConsumerGroup,
Operation::DeleteConsumerGroup,
Operation::CreateUser,
- Operation::UpdateUser,
Operation::DeleteUser,
Operation::ChangePassword,
Operation::UpdatePermissions,
@@ -346,6 +361,7 @@ mod tests {
Operation::DeletePersonalAccessToken,
Operation::JoinConsumerGroup,
Operation::LeaveConsumerGroup,
+ Operation::UpdateResource,
Operation::SendMessages,
Operation::StoreConsumerOffset,
Operation::DeleteConsumerOffset,
@@ -461,4 +477,43 @@ mod tests {
);
}
}
+
+ #[test]
+ fn given_rewrap_operation_when_classifying_should_be_internal_metadata_and_not_client_allowed()
+ {
+ let rewrap = Operation::RewrapStreamKeys;
+ assert_eq!(rewrap as u8, 75);
+ assert!(rewrap.is_internal());
+ assert!(rewrap.is_metadata());
+ assert!(rewrap.is_metadata_plane());
+ assert!(!rewrap.is_client_allowed());
+ assert_eq!(rewrap.to_command_code(), None);
+ assert_eq!(rewrap.request_operation(), rewrap);
+ assert!(!Operation::is_known_code(69));
+ }
+
+ #[test]
+ fn given_resolved_update_operations_when_classifying_should_be_internal_metadata() {
+ for resolved in [
+ Operation::UpdateStreamResolved,
+ Operation::UpdateTopicResolved,
+ Operation::UpdateUserResolved,
+ ] {
+ assert!(resolved.is_internal(), "{resolved:?}");
+ assert!(resolved.is_metadata(), "{resolved:?}");
+ assert!(resolved.is_result_framed(), "{resolved:?}");
+ assert!(!resolved.is_client_allowed(), "{resolved:?}");
+ assert_eq!(resolved.to_command_code(), None);
+ assert_eq!(resolved.request_operation(), Operation::UpdateResource);
+ }
+ assert!(Operation::UpdateResource.is_metadata());
+ assert!(Operation::UpdateResource.is_client_allowed());
+ assert_eq!(Operation::UpdateResource as u8, 150);
+ for retired in [129u8, 133, 142] {
+ assert!(
+ !Operation::is_known_code(retired),
+ "{retired} must stay retired"
+ );
+ }
+ }
}
diff --git a/core/binary_protocol/src/consensus/request.rs b/core/binary_protocol/src/consensus/request.rs
index 48dab128e..02a84468b 100644
--- a/core/binary_protocol/src/consensus/request.rs
+++ b/core/binary_protocol/src/consensus/request.rs
@@ -18,8 +18,8 @@
use crate::codes::{LOGIN_REGISTER_CODE, LOGIN_REGISTER_WITH_PAT_CODE, LOGOUT_USER_CODE};
use crate::consensus::{Command, HEADER_SIZE, Operation, RequestHeader};
use crate::error::WireError;
+use ring::rand::{SecureRandom, SystemRandom};
use std::ops::Range;
-use twox_hash::XxHash3_64;
/// Where `RequestHeader::reserved` carries the legacy command code of a
/// `NonReplicated` request, as little-endian `u32`.
@@ -48,8 +48,8 @@ impl RequestHeader {
/// Every rule that depends only on the code and the payload lives here so
/// that each client, and the fixtures other SDKs verify against, agree on
/// the bytes: metadata and session operations stamp `request_checksum`
- /// with the payload hash, partition and non-replicated operations leave
- /// it zero (send batches carry their own checksum and non-replicated
+ /// with a random per-request identifier, partition and non-replicated
+ /// operations leave it zero (send batches carry their own checksum and non-replicated
/// requests bypass dedup), and a non-replicated request carries its
/// command code in `reserved`.
///
@@ -62,7 +62,8 @@ impl RequestHeader {
/// # Errors
///
/// Returns `WireError::PayloadTooLarge` when the frame would not fit the
- /// `u32` size field.
+ /// `u32` size field, or `WireError::RandomSourceUnavailable` if the secure
+ /// random source fails for a metadata or session request.
pub fn for_request(
code: u32,
client: u128,
@@ -80,7 +81,17 @@ impl RequestHeader {
{
0
} else {
- u128::from(XxHash3_64::oneshot(payload))
+ let random = SystemRandom::new();
+ loop {
+ let mut bytes = [0u8; 16];
+ random
+ .fill(&mut bytes)
+ .map_err(|_| WireError::RandomSourceUnavailable)?;
+ let id = u128::from_le_bytes(bytes);
+ if id != 0 {
+ break id;
+ }
+ }
};
let mut reserved = [0; 4];
if operation == Operation::NonReplicated {
@@ -107,6 +118,7 @@ impl RequestHeader {
mod tests {
use super::*;
use crate::codes::{CREATE_STREAM_CODE, PING_CODE, SEND_MESSAGES_CODE};
+ use twox_hash::XxHash3_64;
#[test]
fn login_codes_register_and_logout_logs_out() {
@@ -120,13 +132,16 @@ mod tests {
}
#[test]
- fn metadata_requests_are_stamped_with_the_payload_hash() {
+ fn given_identical_metadata_bodies_when_building_requests_should_stamp_distinct_random_ids() {
let payload = [1, 2, 3];
let header = RequestHeader::for_request(CREATE_STREAM_CODE, 7, 8, 9, &payload).unwrap();
+ let another = RequestHeader::for_request(CREATE_STREAM_CODE, 7, 9, 9, &payload).unwrap();
assert_eq!(header.command, Command::Request);
assert_eq!(header.size as usize, HEADER_SIZE + payload.len());
assert_eq!((header.client, header.request, header.session), (7, 8, 9));
- assert_eq!(
+ assert_ne!(header.request_checksum, 0);
+ assert_ne!(header.request_checksum, another.request_checksum);
+ assert_ne!(
header.request_checksum,
u128::from(XxHash3_64::oneshot(&payload))
);
diff --git a/core/binary_protocol/src/dispatch.rs b/core/binary_protocol/src/dispatch.rs
index 93be52300..0e809b47d 100644
--- a/core/binary_protocol/src/dispatch.rs
+++ b/core/binary_protocol/src/dispatch.rs
@@ -77,7 +77,6 @@ pub const COMMAND_TABLE: &[CommandMeta] = &[
CommandMeta::non_replicated(GET_USERS_CODE, "user.list"),
CommandMeta::replicated(CREATE_USER_CODE, "user.create", Operation::CreateUser),
CommandMeta::replicated(DELETE_USER_CODE, "user.delete", Operation::DeleteUser),
- CommandMeta::replicated(UPDATE_USER_CODE, "user.update", Operation::UpdateUser),
CommandMeta::replicated(
UPDATE_PERMISSIONS_CODE,
"user.permissions",
@@ -130,13 +129,11 @@ pub const COMMAND_TABLE: &[CommandMeta] = &[
CommandMeta::non_replicated(GET_STREAMS_CODE, "stream.list"),
CommandMeta::replicated(CREATE_STREAM_CODE, "stream.create", Operation::CreateStream),
CommandMeta::replicated(DELETE_STREAM_CODE, "stream.delete", Operation::DeleteStream),
- CommandMeta::replicated(UPDATE_STREAM_CODE, "stream.update", Operation::UpdateStream),
// Topics
CommandMeta::non_replicated(GET_TOPIC_CODE, "topic.get"),
CommandMeta::non_replicated(GET_TOPICS_CODE, "topic.list"),
CommandMeta::replicated(CREATE_TOPIC_CODE, "topic.create", Operation::CreateTopic),
CommandMeta::replicated(DELETE_TOPIC_CODE, "topic.delete", Operation::DeleteTopic),
- CommandMeta::replicated(UPDATE_TOPIC_CODE, "topic.update", Operation::UpdateTopic),
// Partitions
CommandMeta::replicated(
CREATE_PARTITIONS_CODE,
@@ -184,6 +181,12 @@ pub const COMMAND_TABLE: &[CommandMeta] = &[
CommandMeta::non_replicated(DESCRIBE_OPTIONS_CODE, "options.describe"),
CommandMeta::non_replicated(BIND_SESSION_CODE, "session.bind"),
CommandMeta::non_replicated(GET_PARTITION_ROUTE_CODE, "partition.route"),
+ // Generic option patch for streams, topics and users
+ CommandMeta::replicated(
+ UPDATE_RESOURCE_CODE,
+ "resource.update",
+ Operation::UpdateResource,
+ ),
];
/// Lookup command metadata by command code.
@@ -206,45 +209,43 @@ pub const fn lookup_command(code: u32) -> Option<&'static CommandMeta> {
GET_USERS_CODE => 8,
CREATE_USER_CODE => 9,
DELETE_USER_CODE => 10,
- UPDATE_USER_CODE => 11,
- UPDATE_PERMISSIONS_CODE => 12,
- CHANGE_PASSWORD_CODE => 13,
- LOGIN_USER_CODE => 14,
- LOGOUT_USER_CODE => 15,
- LOGIN_REGISTER_CODE => 16,
- GET_PERSONAL_ACCESS_TOKENS_CODE => 17,
- CREATE_PERSONAL_ACCESS_TOKEN_CODE => 18,
- DELETE_PERSONAL_ACCESS_TOKEN_CODE => 19,
- LOGIN_WITH_PERSONAL_ACCESS_TOKEN_CODE => 20,
- POLL_MESSAGES_CODE => 21,
- SEND_MESSAGES_CODE => 22,
- GET_CONSUMER_OFFSET_CODE => 23,
- STORE_CONSUMER_OFFSET_CODE => 24,
- DELETE_CONSUMER_OFFSET_CODE => 25,
- GET_STREAM_CODE => 26,
- GET_STREAMS_CODE => 27,
- CREATE_STREAM_CODE => 28,
- DELETE_STREAM_CODE => 29,
- UPDATE_STREAM_CODE => 30,
- GET_TOPIC_CODE => 31,
- GET_TOPICS_CODE => 32,
- CREATE_TOPIC_CODE => 33,
- DELETE_TOPIC_CODE => 34,
- UPDATE_TOPIC_CODE => 35,
- CREATE_PARTITIONS_CODE => 36,
- DELETE_PARTITIONS_CODE => 37,
- DELETE_SEGMENTS_CODE => 38,
- GET_CONSUMER_GROUP_CODE => 39,
- GET_CONSUMER_GROUPS_CODE => 40,
- CREATE_CONSUMER_GROUP_CODE => 41,
- DELETE_CONSUMER_GROUP_CODE => 42,
- JOIN_CONSUMER_GROUP_CODE => 43,
- LEAVE_CONSUMER_GROUP_CODE => 44,
- SYNC_CONSUMER_GROUP_CODE => 45,
- LOGIN_REGISTER_WITH_PAT_CODE => 46,
- DESCRIBE_OPTIONS_CODE => 47,
- BIND_SESSION_CODE => 48,
- GET_PARTITION_ROUTE_CODE => 49,
+ UPDATE_PERMISSIONS_CODE => 11,
+ CHANGE_PASSWORD_CODE => 12,
+ LOGIN_USER_CODE => 13,
+ LOGOUT_USER_CODE => 14,
+ LOGIN_REGISTER_CODE => 15,
+ GET_PERSONAL_ACCESS_TOKENS_CODE => 16,
+ CREATE_PERSONAL_ACCESS_TOKEN_CODE => 17,
+ DELETE_PERSONAL_ACCESS_TOKEN_CODE => 18,
+ LOGIN_WITH_PERSONAL_ACCESS_TOKEN_CODE => 19,
+ POLL_MESSAGES_CODE => 20,
+ SEND_MESSAGES_CODE => 21,
+ GET_CONSUMER_OFFSET_CODE => 22,
+ STORE_CONSUMER_OFFSET_CODE => 23,
+ DELETE_CONSUMER_OFFSET_CODE => 24,
+ GET_STREAM_CODE => 25,
+ GET_STREAMS_CODE => 26,
+ CREATE_STREAM_CODE => 27,
+ DELETE_STREAM_CODE => 28,
+ GET_TOPIC_CODE => 29,
+ GET_TOPICS_CODE => 30,
+ CREATE_TOPIC_CODE => 31,
+ DELETE_TOPIC_CODE => 32,
+ CREATE_PARTITIONS_CODE => 33,
+ DELETE_PARTITIONS_CODE => 34,
+ DELETE_SEGMENTS_CODE => 35,
+ GET_CONSUMER_GROUP_CODE => 36,
+ GET_CONSUMER_GROUPS_CODE => 37,
+ CREATE_CONSUMER_GROUP_CODE => 38,
+ DELETE_CONSUMER_GROUP_CODE => 39,
+ JOIN_CONSUMER_GROUP_CODE => 40,
+ LEAVE_CONSUMER_GROUP_CODE => 41,
+ SYNC_CONSUMER_GROUP_CODE => 42,
+ LOGIN_REGISTER_WITH_PAT_CODE => 43,
+ DESCRIBE_OPTIONS_CODE => 44,
+ BIND_SESSION_CODE => 45,
+ GET_PARTITION_ROUTE_CODE => 46,
+ UPDATE_RESOURCE_CODE => 47,
_ => return None,
};
Some(&COMMAND_TABLE[idx])
@@ -258,30 +259,32 @@ pub const fn lookup_command(code: u32) -> Option<&'static CommandMeta> {
pub const fn lookup_by_operation(op: Operation) -> Option<&'static CommandMeta> {
// Indices must match the order of entries in COMMAND_TABLE above.
let idx = match op {
- Operation::CreateStream => 28,
- Operation::UpdateStream => 30,
- Operation::DeleteStream => 29,
- Operation::CreateTopic => 33,
- Operation::UpdateTopic => 35,
- Operation::DeleteTopic => 34,
- Operation::CreatePartitions => 36,
- Operation::DeletePartitions => 37,
- Operation::DeleteSegments => 38,
- Operation::CreateConsumerGroup => 41,
- Operation::DeleteConsumerGroup => 42,
- Operation::JoinConsumerGroup => 43,
- Operation::LeaveConsumerGroup => 44,
Operation::CreateUser => 9,
- Operation::UpdateUser => 11,
Operation::DeleteUser => 10,
- Operation::ChangePassword => 13,
- Operation::UpdatePermissions => 12,
- Operation::CreatePersonalAccessToken => 18,
- Operation::DeletePersonalAccessToken => 19,
- Operation::SendMessages => 22,
- Operation::StoreConsumerOffset => 24,
- Operation::DeleteConsumerOffset => 25,
+ Operation::UpdatePermissions => 11,
+ Operation::ChangePassword => 12,
+ Operation::CreatePersonalAccessToken => 17,
+ Operation::DeletePersonalAccessToken => 18,
+ Operation::SendMessages => 21,
+ Operation::StoreConsumerOffset => 23,
+ Operation::DeleteConsumerOffset => 24,
+ Operation::CreateStream => 27,
+ Operation::DeleteStream => 28,
+ Operation::CreateTopic => 31,
+ Operation::DeleteTopic => 32,
+ Operation::CreatePartitions => 33,
+ Operation::DeletePartitions => 34,
+ Operation::DeleteSegments => 35,
+ Operation::CreateConsumerGroup => 38,
+ Operation::DeleteConsumerGroup => 39,
+ Operation::JoinConsumerGroup => 40,
+ Operation::LeaveConsumerGroup => 41,
+ Operation::UpdateResource => 47,
Operation::CreateTopicWithAssignments
+ | Operation::UpdateStreamResolved
+ | Operation::UpdateTopicResolved
+ | Operation::UpdateUserResolved
+ | Operation::RewrapStreamKeys
| Operation::CreatePartitionsWithAssignments
| Operation::RemoveConsumerGroupMember
| Operation::CompleteConsumerGroupRevocation
diff --git a/core/binary_protocol/src/error.rs b/core/binary_protocol/src/error.rs
index 208b15c3f..619afb313 100644
--- a/core/binary_protocol/src/error.rs
+++ b/core/binary_protocol/src/error.rs
@@ -47,6 +47,9 @@ pub enum WireError {
#[error("payload too large: {size} bytes, max {max}")]
PayloadTooLarge { size: usize, max: usize },
+ #[error("secure random source unavailable")]
+ RandomSourceUnavailable,
+
#[error("validation failed: {0}")]
Validation(Cow<'static, str>),
diff --git a/core/binary_protocol/src/primitives/options.rs b/core/binary_protocol/src/primitives/options.rs
index 16e781b2c..bc6be4026 100644
--- a/core/binary_protocol/src/primitives/options.rs
+++ b/core/binary_protocol/src/primitives/options.rs
@@ -26,6 +26,7 @@
//! admission.
use std::borrow::Cow;
+use std::fmt::{Debug, Formatter, Result as FmtResult};
use bytes::{BufMut, Bytes, BytesMut};
@@ -176,9 +177,18 @@ pub fn decode_options_prefixed(buf: &[u8], pos: usize) -> Result<(WireOptions, u
///
/// Like `WireUserHeaders`, this type is opaque at the wire layer. Domain
/// interpretation of value kind codes happens in `iggy_common`.
-#[derive(Debug, Clone, PartialEq, Eq)]
+#[derive(Clone, PartialEq, Eq)]
pub struct WireOptions(Bytes);
+impl Debug for WireOptions {
+ fn fmt(&self, formatter: &mut Formatter<'_>) -> FmtResult {
+ formatter
+ .debug_struct("WireOptions")
+ .field("bytes_len", &self.0.len())
+ .finish()
+ }
+}
+
impl WireOptions {
/// Validate and wrap a borrowed byte slice (copies into new `Bytes`).
///
@@ -270,6 +280,19 @@ mod tests {
buf
}
+ #[test]
+ fn given_stream_key_when_debugging_options_should_hide_key_bytes_and_keep_the_block() {
+ let key = [b'k'; 32];
+ let buf = encode(&[(STRING, b"encryption_key", 1, &key)]);
+ let options = WireOptions::from_slice(&buf).expect("valid stream options");
+
+ let debugged = format!("{options:?}");
+
+ assert!(!debugged.contains("kkkkkkkk"));
+ assert!(!debugged.contains("encryption_key"));
+ assert_eq!(options.iter().next().expect("key entry").value, key);
+ }
+
#[test]
fn empty_buffer_is_zero_options() {
assert_eq!(validate_options(&[]).unwrap(), 0);
diff --git a/core/binary_protocol/src/requests/streams/mod.rs b/core/binary_protocol/src/requests/streams/mod.rs
index 837da7fc2..0eb77654e 100644
--- a/core/binary_protocol/src/requests/streams/mod.rs
+++ b/core/binary_protocol/src/requests/streams/mod.rs
@@ -19,10 +19,15 @@ pub mod create_stream;
pub mod delete_stream;
pub mod get_stream;
pub mod get_streams;
-pub mod update_stream;
+pub mod rewrap_stream_keys;
+pub mod update_stream_resolved;
pub use create_stream::CreateStreamRequest;
pub use delete_stream::DeleteStreamRequest;
pub use get_stream::GetStreamRequest;
pub use get_streams::GetStreamsRequest;
-pub use update_stream::UpdateStreamRequest;
+pub use rewrap_stream_keys::{
+ MAX_WRAP_SET_ENVELOPES, RewrapStreamKeyEntry, RewrapStreamKeysRequest,
+ WRAPPED_STREAM_KEY_LENGTH,
+};
+pub use update_stream_resolved::UpdateStreamResolvedRequest;
diff --git a/core/binary_protocol/src/requests/streams/rewrap_stream_keys.rs b/core/binary_protocol/src/requests/streams/rewrap_stream_keys.rs
new file mode 100644
index 000000000..68548c6d1
--- /dev/null
+++ b/core/binary_protocol/src/requests/streams/rewrap_stream_keys.rs
@@ -0,0 +1,216 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::WireError;
+use crate::codec::{WireDecode, WireEncode, read_bytes, read_u8, read_u32_le, read_u64_le};
+use bytes::{BufMut, Bytes, BytesMut};
+use std::borrow::Cow;
+use std::fmt::{Debug, Formatter};
+
+/// Bytes of one wrapped stream key envelope.
+pub const WRAPPED_STREAM_KEY_LENGTH: usize = 78;
+/// Most envelopes one wrap set holds.
+pub const MAX_WRAP_SET_ENVELOPES: usize = 3;
+const KEY_FINGERPRINT_LENGTH: usize = 8;
+const ENTRY_FIXED_SIZE: usize = 4 + 8 + KEY_FINGERPRINT_LENGTH + 1;
+
+/// Server-originated: replace the wrap sets of encrypted streams after a
+/// key-encryption key rotation. Apply checks each entry against the stored
+/// stream and skips one that no longer matches.
+///
+/// Wire format:
+/// ```text
+/// [count:u32]([stream_id:u32][created_op:u64][key_fingerprint:8][set_len:u8][set])*
+/// ```
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RewrapStreamKeysRequest {
+ pub entries: Vec<RewrapStreamKeyEntry>,
+}
+
+/// One stream incarnation and the wrap set it should store.
+#[derive(Clone, PartialEq, Eq)]
+pub struct RewrapStreamKeyEntry {
+ pub stream_id: u32,
+ pub created_op: u64,
+ pub key_fingerprint: [u8; KEY_FINGERPRINT_LENGTH],
+ pub wrap_set: Bytes,
+}
+
+impl Debug for RewrapStreamKeyEntry {
+ /// Prints the envelope count, never the wrapped key bytes.
+ fn fmt(&self, formatter: &mut Formatter<'_>) -> std::fmt::Result {
+ formatter
+ .debug_struct("RewrapStreamKeyEntry")
+ .field("stream_id", &self.stream_id)
+ .field("created_op", &self.created_op)
+ .field("key_fingerprint", &self.key_fingerprint)
+ .field(
+ "envelopes",
+ &(self.wrap_set.len() / WRAPPED_STREAM_KEY_LENGTH),
+ )
+ .finish()
+ }
+}
+
+impl WireEncode for RewrapStreamKeysRequest {
+ fn encoded_size(&self) -> usize {
+ 4 + self
+ .entries
+ .iter()
+ .map(|entry| ENTRY_FIXED_SIZE + entry.wrap_set.len())
+ .sum::<usize>()
+ }
+
+ #[allow(clippy::cast_possible_truncation)]
+ fn encode(&self, buf: &mut BytesMut) {
+ buf.put_u32_le(self.entries.len() as u32);
+ for entry in &self.entries {
+ buf.put_u32_le(entry.stream_id);
+ buf.put_u64_le(entry.created_op);
+ buf.put_slice(&entry.key_fingerprint);
+ buf.put_u8(entry.wrap_set.len() as u8);
+ buf.put_slice(&entry.wrap_set);
+ }
+ }
+}
+
+impl WireDecode for RewrapStreamKeysRequest {
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let count = read_u32_le(buf, 0)? as usize;
+ let mut position = 4;
+ // Each entry takes at least its fixed part plus one envelope, so a
+ // count the body cannot hold is refused before allocating for it.
+ let minimum = ENTRY_FIXED_SIZE + WRAPPED_STREAM_KEY_LENGTH;
+ if count > buf.len().saturating_sub(position) / minimum {
+ return Err(validation("rewrap entry count exceeds the body"));
+ }
+ let mut entries = Vec::with_capacity(count);
+ for _ in 0..count {
+ let stream_id = read_u32_le(buf, position)?;
+ let created_op = read_u64_le(buf, position + 4)?;
+ let fingerprint = read_bytes(buf, position + 12, KEY_FINGERPRINT_LENGTH)?;
+ let set_length = usize::from(read_u8(buf, position + 20)?);
+ if set_length == 0
+ || !set_length.is_multiple_of(WRAPPED_STREAM_KEY_LENGTH)
+ || set_length > MAX_WRAP_SET_ENVELOPES * WRAPPED_STREAM_KEY_LENGTH
+ {
+ return Err(validation(
+ "rewrap set length is not one to three envelopes",
+ ));
+ }
+ let wrap_set = read_bytes(buf, position + ENTRY_FIXED_SIZE, set_length)?;
+ let mut key_fingerprint = [0u8; KEY_FINGERPRINT_LENGTH];
+ key_fingerprint.copy_from_slice(fingerprint);
+ entries.push(RewrapStreamKeyEntry {
+ stream_id,
+ created_op,
+ key_fingerprint,
+ wrap_set: Bytes::copy_from_slice(wrap_set),
+ });
+ position += ENTRY_FIXED_SIZE + set_length;
+ }
+ if position != buf.len() {
+ return Err(validation("trailing bytes after the rewrap entries"));
+ }
+ Ok((Self { entries }, position))
+ }
+}
+
+const fn validation(reason: &'static str) -> WireError {
+ WireError::Validation(Cow::Borrowed(reason))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn entry(stream_id: u32, envelopes: usize) -> RewrapStreamKeyEntry {
+ RewrapStreamKeyEntry {
+ stream_id,
+ created_op: 7,
+ key_fingerprint: [0xAB; KEY_FINGERPRINT_LENGTH],
+ wrap_set: Bytes::from(vec![0x5A; envelopes * WRAPPED_STREAM_KEY_LENGTH]),
+ }
+ }
+
+ #[test]
+ fn given_rewrap_entries_when_roundtripping_should_match_golden_bytes() {
+ let single = RewrapStreamKeysRequest {
+ entries: vec![entry(3, 1)],
+ };
+ let bytes = single.to_bytes();
+ let mut golden = vec![1, 0, 0, 0, 3, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0];
+ golden.extend_from_slice(&[0xAB; KEY_FINGERPRINT_LENGTH]);
+ golden.push(78);
+ golden.extend_from_slice(&[0x5A; WRAPPED_STREAM_KEY_LENGTH]);
+ assert_eq!(bytes.as_ref(), golden.as_slice());
+ assert_eq!(single.encoded_size(), golden.len());
+
+ let several = RewrapStreamKeysRequest {
+ entries: vec![entry(1, 2), entry(2, 3), entry(9, 1)],
+ };
+ let bytes = several.to_bytes();
+ let (decoded, consumed) = RewrapStreamKeysRequest::decode(&bytes).unwrap();
+ assert_eq!(consumed, bytes.len());
+ assert_eq!(decoded, several);
+ }
+
+ #[test]
+ fn given_malformed_rewrap_body_when_decoding_should_reject_without_panicking() {
+ let bytes = RewrapStreamKeysRequest {
+ entries: vec![entry(1, 1), entry(2, 2)],
+ }
+ .to_bytes();
+ for length in 0..bytes.len() {
+ assert!(
+ RewrapStreamKeysRequest::decode(&bytes[..length]).is_err(),
+ "a body cut at {length} must be refused"
+ );
+ }
+ let mut trailing = bytes.to_vec();
+ trailing.push(0);
+ assert!(RewrapStreamKeysRequest::decode(&trailing).is_err());
+
+ let set_length_at = 4 + 4 + 8 + KEY_FINGERPRINT_LENGTH;
+ for invalid in [0u8, 77, 255] {
+ let mut body = RewrapStreamKeysRequest {
+ entries: vec![entry(1, 1)],
+ }
+ .to_bytes()
+ .to_vec();
+ body[set_length_at] = invalid;
+ assert!(RewrapStreamKeysRequest::decode(&body).is_err(), "{invalid}");
+ }
+ let oversized_count = [0xFF, 0xFF, 0xFF, 0xFF];
+ assert!(RewrapStreamKeysRequest::decode(&oversized_count).is_err());
+ }
+
+ #[test]
+ fn given_rewrap_request_when_debugging_should_not_print_wrapped_keys() {
+ let rendered = format!(
+ "{:?}",
+ RewrapStreamKeysRequest {
+ entries: vec![entry(1, 2)],
+ }
+ );
+ assert!(
+ !rendered.contains("ZZ") && !rendered.contains("90"),
+ "{rendered}"
+ );
+ assert!(rendered.contains("envelopes: 2"), "{rendered}");
+ }
+}
diff --git a/core/binary_protocol/src/requests/streams/update_stream.rs b/core/binary_protocol/src/requests/streams/update_stream.rs
deleted file mode 100644
index d36fadd16..000000000
--- a/core/binary_protocol/src/requests/streams/update_stream.rs
+++ /dev/null
@@ -1,127 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-use crate::WireError;
-use crate::WireIdentifier;
-use crate::codec::{WireDecode, WireEncode};
-use crate::primitives::identifier::WireName;
-use crate::primitives::options::WireOptions;
-use bytes::BytesMut;
-
-/// `UpdateStream` request.
-///
-/// Wire format: `[identifier][name_len:1][name:N][options TLV to end]`
-///
-/// The options block mirrors `CreateStream`'s, so a stream setting added to
-/// the catalog is updatable without another layout change. Keys absent from
-/// the block are left alone rather than reset: an update patches the stored
-/// map, so a client built before a key existed cannot erase it.
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct UpdateStreamRequest {
- pub stream_id: WireIdentifier,
- pub name: WireName,
- pub options: WireOptions,
-}
-
-impl WireEncode for UpdateStreamRequest {
- fn encoded_size(&self) -> usize {
- self.stream_id.encoded_size() + self.name.encoded_size() + self.options.encoded_size()
- }
-
- fn encode(&self, buf: &mut BytesMut) {
- self.stream_id.encode(buf);
- self.name.encode(buf);
- self.options.encode(buf);
- }
-}
-
-impl WireDecode for UpdateStreamRequest {
- fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
- let (stream_id, mut pos) = WireIdentifier::decode(buf)?;
- let (name, consumed) = WireName::decode(&buf[pos..])?;
- pos += consumed;
- let options = WireOptions::from_slice(&buf[pos..])?;
- Ok((
- Self {
- stream_id,
- name,
- options,
- },
- buf.len(),
- ))
- }
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
- use crate::primitives::user_headers::encode_user_headers;
-
- fn sample_options() -> WireOptions {
- let mut buf = BytesMut::new();
- encode_user_headers(&[(2, b"future_key", 2, b"value")], &mut buf);
- WireOptions::from_bytes(buf.freeze()).unwrap()
- }
-
- #[test]
- fn roundtrip() {
- let req = UpdateStreamRequest {
- stream_id: WireIdentifier::named("old-name").unwrap(),
- name: WireName::new("new-name").unwrap(),
- options: sample_options(),
- };
- let bytes = req.to_bytes();
- let (decoded, consumed) = UpdateStreamRequest::decode(&bytes).unwrap();
- assert_eq!(consumed, bytes.len());
- assert_eq!(decoded, req);
- }
-
- #[test]
- fn truncated_fixed_fields_return_error() {
- let req = UpdateStreamRequest {
- stream_id: WireIdentifier::numeric(1),
- name: WireName::new("test").unwrap(),
- options: WireOptions::empty(),
- };
- let bytes = req.to_bytes();
- for i in 0..bytes.len() {
- assert!(
- UpdateStreamRequest::decode(&bytes[..i]).is_err(),
- "expected error for truncation at byte {i}"
- );
- }
- }
-
- #[test]
- fn truncated_options_return_error() {
- // One pair, so any strict-interior truncation is invalid. Cutting at
- // the block boundary is a legitimate update carrying no options.
- let req = UpdateStreamRequest {
- stream_id: WireIdentifier::numeric(1),
- name: WireName::new("test").unwrap(),
- options: sample_options(),
- };
- let bytes = req.to_bytes();
- let fixed_end = bytes.len() - req.options.encoded_size();
- for i in fixed_end + 1..bytes.len() {
- assert!(
- UpdateStreamRequest::decode(&bytes[..i]).is_err(),
- "expected error for truncation at byte {i}"
- );
- }
- }
-}
diff --git a/core/binary_protocol/src/requests/streams/update_stream_resolved.rs b/core/binary_protocol/src/requests/streams/update_stream_resolved.rs
new file mode 100644
index 000000000..7db126331
--- /dev/null
+++ b/core/binary_protocol/src/requests/streams/update_stream_resolved.rs
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::WireError;
+use crate::WireIdentifier;
+use crate::codec::{WireDecode, WireEncode};
+use crate::primitives::identifier::WireName;
+use crate::primitives::options::WireOptions;
+use crate::requests::system::update_resource::{
+ decode_optional_name, decode_resolved_blocks, encode_optional_name, encode_resolved_blocks,
+ optional_name_size, resolved_blocks_size,
+};
+use bytes::BytesMut;
+
+/// An `UpdateResource` on a stream after prepare: the rename extracted, the
+/// set keys canonical in `explicit_options`, the reset keys resolved to their
+/// defaults in `derived_options`.
+///
+/// Wire format:
+/// ```text
+/// [stream_id][has_name:u8][name?][explicit_len:u32][explicit][derived_len:u32][derived]
+/// ```
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct UpdateStreamResolvedRequest {
+ pub stream_id: WireIdentifier,
+ pub name: Option<WireName>,
+ pub explicit_options: WireOptions,
+ pub derived_options: WireOptions,
+}
+
+impl WireEncode for UpdateStreamResolvedRequest {
+ fn encoded_size(&self) -> usize {
+ self.stream_id.encoded_size()
+ + optional_name_size(self.name.as_ref())
+ + resolved_blocks_size(&self.explicit_options, &self.derived_options)
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ self.stream_id.encode(buf);
+ encode_optional_name(self.name.as_ref(), buf);
+ encode_resolved_blocks(&self.explicit_options, &self.derived_options, buf);
+ }
+}
+
+impl WireDecode for UpdateStreamResolvedRequest {
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let (stream_id, mut position) = WireIdentifier::decode(buf)?;
+ let (name, consumed) = decode_optional_name(buf, position)?;
+ position += consumed;
+ let (explicit_options, derived_options) = decode_resolved_blocks(buf, position)?;
+ Ok((
+ Self {
+ stream_id,
+ name,
+ explicit_options,
+ derived_options,
+ },
+ buf.len(),
+ ))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::primitives::user_headers::encode_user_headers;
+
+ #[test]
+ fn given_resolved_patch_when_roundtripping_should_preserve_columns_and_provenance() {
+ for name in [None, Some(WireName::new("renamed").unwrap())] {
+ let request = UpdateStreamResolvedRequest {
+ stream_id: WireIdentifier::named("events").unwrap(),
+ name,
+ explicit_options: options(b"future_key", b"value"),
+ derived_options: WireOptions::empty(),
+ };
+ let bytes = request.to_bytes();
+ assert_eq!(bytes.len(), request.encoded_size());
+ assert_eq!(
+ UpdateStreamResolvedRequest::decode_from(&bytes).unwrap(),
+ request
+ );
+ for length in 0..bytes.len() {
+ assert!(UpdateStreamResolvedRequest::decode(&bytes[..length]).is_err());
+ }
+ let mut trailing = bytes.to_vec();
+ trailing.push(0);
+ assert!(UpdateStreamResolvedRequest::decode(&trailing).is_err());
+ }
+ let mut bad_presence = WireIdentifier::numeric(1).to_bytes().to_vec();
+ bad_presence.push(2);
+ assert!(UpdateStreamResolvedRequest::decode(&bad_presence).is_err());
+ }
+
+ fn options(key: &[u8], value: &[u8]) -> WireOptions {
+ let mut buf = BytesMut::new();
+ encode_user_headers(&[(2, key, 2, value)], &mut buf);
+ WireOptions::from_bytes(buf.freeze()).unwrap()
+ }
+}
diff --git a/core/binary_protocol/src/requests/system/mod.rs b/core/binary_protocol/src/requests/system/mod.rs
index 9be7a86c2..9d5c568c8 100644
--- a/core/binary_protocol/src/requests/system/mod.rs
+++ b/core/binary_protocol/src/requests/system/mod.rs
@@ -26,6 +26,7 @@ pub mod get_snapshot;
pub mod get_stats;
pub mod ping;
pub mod retire_sessions;
+pub mod update_resource;
pub use bind_session::{BindSessionRequest, SessionIdentity};
pub use describe_options::{
@@ -40,3 +41,4 @@ pub use get_snapshot::GetSnapshotRequest;
pub use get_stats::GetStatsRequest;
pub use ping::PingRequest;
pub use retire_sessions::{MAX_SESSIONS_PER_RETIREMENT, RetireSessionsRequest};
+pub use update_resource::{MAX_RESET_KEYS, ResourceTarget, UpdateResourceRequest, WireResetKeys};
diff --git a/core/binary_protocol/src/requests/system/update_resource.rs b/core/binary_protocol/src/requests/system/update_resource.rs
new file mode 100644
index 000000000..799e63452
--- /dev/null
+++ b/core/binary_protocol/src/requests/system/update_resource.rs
@@ -0,0 +1,476 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use super::describe_options::{OPTIONS_SCOPE_STREAM, OPTIONS_SCOPE_TOPIC, OPTIONS_SCOPE_USER};
+use crate::WireError;
+use crate::WireIdentifier;
+use crate::codec::{WireDecode, WireEncode, read_u8};
+use crate::primitives::identifier::WireName;
+use crate::primitives::options::{
+ WireOptions, decode_options_prefixed, encode_options_prefixed, options_prefixed_size,
+};
+use bytes::{BufMut, BytesMut};
+use std::borrow::Cow;
+use std::collections::BTreeSet;
+use std::ops::Deref;
+use std::slice::Iter;
+
+/// Most reset names one patch carries, the reach of its count byte.
+pub const MAX_RESET_KEYS: usize = u8::MAX as usize;
+
+/// The resource an `UpdateResource` patches, tagged by its options scope code.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum ResourceTarget {
+ Topic {
+ stream_id: WireIdentifier,
+ topic_id: WireIdentifier,
+ },
+ Stream {
+ stream_id: WireIdentifier,
+ },
+ User {
+ user_id: WireIdentifier,
+ },
+}
+
+/// Patch one resource's options: set the keys in `options`, reset the keys in
+/// `reset` to their defaults, leave every other key alone.
+///
+/// Wire format:
+/// ```text
+/// [scope:u8][identifiers][reset_count:u8][reset_key:WireName]*[options to end of body]
+/// ```
+/// Identifiers are `[stream_id][topic_id]` for a topic, `[stream_id]` for a
+/// stream and `[user_id]` for a user. A patch with nothing to set or reset, a
+/// key named twice, or a key both set and reset is refused at decode.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct UpdateResourceRequest {
+ pub target: ResourceTarget,
+ pub reset: WireResetKeys,
+ pub options: WireOptions,
+}
+
+/// The names a patch resets, at most [`MAX_RESET_KEYS`] so the count byte
+/// always matches the names after it.
+///
+/// Wire format: `[count:u8][name:WireName]*`
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct WireResetKeys(Vec<WireName>);
+
+impl ResourceTarget {
+ /// The options scope code the target is tagged with on the wire.
+ #[must_use]
+ pub const fn scope_code(&self) -> u8 {
+ match self {
+ Self::Topic { .. } => OPTIONS_SCOPE_TOPIC,
+ Self::Stream { .. } => OPTIONS_SCOPE_STREAM,
+ Self::User { .. } => OPTIONS_SCOPE_USER,
+ }
+ }
+}
+
+impl TryFrom<Vec<WireName>> for WireResetKeys {
+ type Error = WireError;
+
+ fn try_from(names: Vec<WireName>) -> Result<Self, Self::Error> {
+ if names.len() > MAX_RESET_KEYS {
+ return Err(validation("more reset keys than one patch carries"));
+ }
+ Ok(Self(names))
+ }
+}
+
+impl Deref for WireResetKeys {
+ type Target = [WireName];
+
+ fn deref(&self) -> &Self::Target {
+ &self.0
+ }
+}
+
+impl<'a> IntoIterator for &'a WireResetKeys {
+ type Item = &'a WireName;
+ type IntoIter = Iter<'a, WireName>;
+
+ fn into_iter(self) -> Self::IntoIter {
+ self.0.iter()
+ }
+}
+
+impl WireEncode for WireResetKeys {
+ fn encoded_size(&self) -> usize {
+ 1 + self.0.iter().map(WireEncode::encoded_size).sum::<usize>()
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ // The constructor bounds the length to what the count byte holds.
+ #[allow(clippy::cast_possible_truncation)]
+ buf.put_u8(self.0.len() as u8);
+ for name in &self.0 {
+ name.encode(buf);
+ }
+ }
+}
+
+impl WireDecode for WireResetKeys {
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let count = read_u8(buf, 0)?;
+ let mut position = 1;
+ let mut names = Vec::with_capacity(usize::from(count));
+ let mut seen = BTreeSet::new();
+ for _ in 0..count {
+ let (name, consumed) = WireName::decode(&buf[position..])?;
+ position += consumed;
+ if !seen.insert(name.as_str().to_owned()) {
+ return Err(validation("a reset key is named twice"));
+ }
+ names.push(name);
+ }
+ Ok((Self(names), position))
+ }
+}
+
+impl WireEncode for ResourceTarget {
+ fn encoded_size(&self) -> usize {
+ 1 + match self {
+ Self::Topic {
+ stream_id,
+ topic_id,
+ } => stream_id.encoded_size() + topic_id.encoded_size(),
+ Self::Stream { stream_id } => stream_id.encoded_size(),
+ Self::User { user_id } => user_id.encoded_size(),
+ }
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ buf.put_u8(self.scope_code());
+ match self {
+ Self::Topic {
+ stream_id,
+ topic_id,
+ } => {
+ stream_id.encode(buf);
+ topic_id.encode(buf);
+ }
+ Self::Stream { stream_id } => stream_id.encode(buf),
+ Self::User { user_id } => user_id.encode(buf),
+ }
+ }
+}
+
+impl WireDecode for ResourceTarget {
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let scope = read_u8(buf, 0)?;
+ let mut position = 1;
+ let target = match scope {
+ OPTIONS_SCOPE_TOPIC => {
+ let (stream_id, consumed) = WireIdentifier::decode(&buf[position..])?;
+ position += consumed;
+ let (topic_id, consumed) = WireIdentifier::decode(&buf[position..])?;
+ position += consumed;
+ Self::Topic {
+ stream_id,
+ topic_id,
+ }
+ }
+ OPTIONS_SCOPE_STREAM => {
+ let (stream_id, consumed) = WireIdentifier::decode(&buf[position..])?;
+ position += consumed;
+ Self::Stream { stream_id }
+ }
+ OPTIONS_SCOPE_USER => {
+ let (user_id, consumed) = WireIdentifier::decode(&buf[position..])?;
+ position += consumed;
+ Self::User { user_id }
+ }
+ unknown => {
+ return Err(WireError::Validation(Cow::Owned(format!(
+ "unknown resource scope: {unknown}"
+ ))));
+ }
+ };
+ Ok((target, position))
+ }
+}
+
+impl WireEncode for UpdateResourceRequest {
+ fn encoded_size(&self) -> usize {
+ self.target.encoded_size() + self.reset.encoded_size() + self.options.encoded_size()
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ self.target.encode(buf);
+ self.reset.encode(buf);
+ self.options.encode(buf);
+ }
+}
+
+impl WireDecode for UpdateResourceRequest {
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let (target, mut position) = ResourceTarget::decode(buf)?;
+ let (reset, consumed) = WireResetKeys::decode(&buf[position..])?;
+ position += consumed;
+ let options = WireOptions::from_slice(&buf[position..])?;
+ for entry in &options {
+ if reset
+ .iter()
+ .any(|name| name.as_str().as_bytes() == entry.key)
+ {
+ return Err(validation("a key is both set and reset"));
+ }
+ }
+ if reset.is_empty() && options.is_empty() {
+ return Err(validation("the patch sets and resets nothing"));
+ }
+ Ok((
+ Self {
+ target,
+ reset,
+ options,
+ },
+ buf.len(),
+ ))
+ }
+}
+
+const fn validation(reason: &'static str) -> WireError {
+ WireError::Validation(Cow::Borrowed(reason))
+}
+
+/// Size of an optional name behind a presence byte.
+pub(crate) fn optional_name_size(name: Option<&WireName>) -> usize {
+ 1 + name.map_or(0, WireEncode::encoded_size)
+}
+
+/// Write an optional name behind a strict 0/1 presence byte.
+pub(crate) fn encode_optional_name(name: Option<&WireName>, buf: &mut BytesMut) {
+ match name {
+ Some(name) => {
+ buf.put_u8(1);
+ name.encode(buf);
+ }
+ None => buf.put_u8(0),
+ }
+}
+
+/// Read an optional name behind a strict 0/1 presence byte at `position`.
+pub(crate) fn decode_optional_name(
+ buf: &[u8],
+ position: usize,
+) -> Result<(Option<WireName>, usize), WireError> {
+ match read_u8(buf, position)? {
+ 0 => Ok((None, 1)),
+ 1 => {
+ let (name, consumed) = WireName::decode(&buf[position + 1..])?;
+ Ok((Some(name), 1 + consumed))
+ }
+ _ => Err(validation("presence byte must be 0 or 1")),
+ }
+}
+
+/// Size of the explicit and derived length-prefixed option blocks.
+pub(crate) fn resolved_blocks_size(explicit: &WireOptions, derived: &WireOptions) -> usize {
+ options_prefixed_size(explicit) + options_prefixed_size(derived)
+}
+
+/// Write the explicit then derived length-prefixed option blocks.
+pub(crate) fn encode_resolved_blocks(
+ explicit: &WireOptions,
+ derived: &WireOptions,
+ buf: &mut BytesMut,
+) {
+ encode_options_prefixed(explicit, buf);
+ encode_options_prefixed(derived, buf);
+}
+
+/// Read the explicit then derived blocks at `position`, which must end the
+/// body exactly.
+pub(crate) fn decode_resolved_blocks(
+ buf: &[u8],
+ position: usize,
+) -> Result<(WireOptions, WireOptions), WireError> {
+ let (explicit, consumed) = decode_options_prefixed(buf, position)?;
+ let (derived, derived_consumed) = decode_options_prefixed(buf, position + consumed)?;
+ if position + consumed + derived_consumed != buf.len() {
+ return Err(validation(
+ "trailing bytes after the resolved option blocks",
+ ));
+ }
+ Ok((explicit, derived))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::primitives::user_headers::encode_user_headers;
+
+ const STRING: u8 = 2;
+ const UINT64: u8 = 12;
+
+ #[test]
+ fn given_resource_patch_when_roundtripping_should_match_golden_bytes() {
+ let topic = UpdateResourceRequest {
+ target: ResourceTarget::Topic {
+ stream_id: WireIdentifier::numeric(1),
+ topic_id: WireIdentifier::named("orders").unwrap(),
+ },
+ reset: reset_keys(&["max_topic_size"]),
+ options: options(&[
+ (
+ STRING,
+ b"message_expiry",
+ UINT64,
+ &86_400_000_000u64.to_le_bytes(),
+ ),
+ (STRING, b"name", STRING, b"orders-v2"),
+ ]),
+ };
+ let bytes = topic.to_bytes();
+ assert_eq!(&bytes[..1], &[OPTIONS_SCOPE_TOPIC]);
+ let (decoded, consumed) = UpdateResourceRequest::decode(&bytes).unwrap();
+ assert_eq!(consumed, bytes.len());
+ assert_eq!(decoded, topic);
+ assert_eq!(decoded.target.scope_code(), OPTIONS_SCOPE_TOPIC);
+
+ let stream = UpdateResourceRequest {
+ target: ResourceTarget::Stream {
+ stream_id: WireIdentifier::numeric(3),
+ },
+ reset: WireResetKeys::default(),
+ options: options(&[(STRING, b"name", STRING, b"renamed")]),
+ };
+ let bytes = stream.to_bytes();
+ let mut expected = vec![OPTIONS_SCOPE_STREAM];
+ expected.extend_from_slice(&WireIdentifier::numeric(3).to_bytes());
+ expected.push(0);
+ expected.extend_from_slice(stream.options.as_bytes());
+ assert_eq!(&bytes[..], expected.as_slice());
+ assert_eq!(UpdateResourceRequest::decode_from(&bytes).unwrap(), stream);
+
+ let user = UpdateResourceRequest {
+ target: ResourceTarget::User {
+ user_id: WireIdentifier::named("alice").unwrap(),
+ },
+ reset: WireResetKeys::default(),
+ options: options(&[
+ (STRING, b"status", STRING, b"inactive"),
+ (STRING, b"username", STRING, b"alice2"),
+ ]),
+ };
+ let bytes = user.to_bytes();
+ assert_eq!(UpdateResourceRequest::decode_from(&bytes).unwrap(), user);
+ }
+
+ #[test]
+ fn given_malformed_patch_when_decoding_should_reject_without_panicking() {
+ let valid = UpdateResourceRequest {
+ target: ResourceTarget::Topic {
+ stream_id: WireIdentifier::numeric(1),
+ topic_id: WireIdentifier::numeric(2),
+ },
+ reset: reset_keys(&["max_topic_size"]),
+ options: options(&[(STRING, b"message_expiry", UINT64, &7u64.to_le_bytes())]),
+ };
+ let bytes = valid.to_bytes();
+ let options_start = bytes.len() - valid.options.encoded_size();
+ // Every prefix that cuts the fixed part or splits a TLV entry fails.
+ // Cutting exactly at the start of the options block leaves a valid
+ // reset-only patch, so that one boundary decodes.
+ for length in 0..bytes.len() {
+ let outcome = UpdateResourceRequest::decode(&bytes[..length]);
+ if length == options_start {
+ assert!(outcome.is_ok(), "reset-only prefix must decode");
+ } else {
+ assert!(outcome.is_err(), "prefix of {length} bytes must fail");
+ }
+ }
+
+ let mut unknown_scope = bytes.to_vec();
+ unknown_scope[0] = 9;
+ assert!(UpdateResourceRequest::decode(&unknown_scope).is_err());
+
+ let duplicate_reset = UpdateResourceRequest {
+ reset: reset_keys(&["max_topic_size", "max_topic_size"]),
+ options: WireOptions::empty(),
+ ..valid.clone()
+ };
+ assert!(UpdateResourceRequest::decode(&duplicate_reset.to_bytes()).is_err());
+
+ let cross_duplicate = UpdateResourceRequest {
+ reset: reset_keys(&["message_expiry"]),
+ ..valid.clone()
+ };
+ assert!(UpdateResourceRequest::decode(&cross_duplicate.to_bytes()).is_err());
+
+ let empty = UpdateResourceRequest {
+ reset: WireResetKeys::default(),
+ options: WireOptions::empty(),
+ ..valid.clone()
+ };
+ assert!(UpdateResourceRequest::decode(&empty.to_bytes()).is_err());
+
+ let mut duplicate_set = valid.target.to_bytes().to_vec();
+ duplicate_set.push(0);
+ let mut entries = BytesMut::new();
+ encode_user_headers(
+ &[
+ (STRING, b"message_expiry", UINT64, &7u64.to_le_bytes()),
+ (STRING, b"message_expiry", UINT64, &8u64.to_le_bytes()),
+ ],
+ &mut entries,
+ );
+ duplicate_set.extend_from_slice(&entries);
+ assert!(UpdateResourceRequest::decode(&duplicate_set).is_err());
+ }
+
+ #[test]
+ fn given_more_reset_names_than_the_count_byte_holds_when_building_should_refuse() {
+ let names = |count: usize| -> Vec<WireName> {
+ (0..count)
+ .map(|index| WireName::new(format!("key_{index}")).unwrap())
+ .collect()
+ };
+ assert!(WireResetKeys::try_from(names(MAX_RESET_KEYS + 1)).is_err());
+ let request = UpdateResourceRequest {
+ target: ResourceTarget::Stream {
+ stream_id: WireIdentifier::numeric(1),
+ },
+ reset: WireResetKeys::try_from(names(MAX_RESET_KEYS)).unwrap(),
+ options: WireOptions::empty(),
+ };
+ assert_eq!(
+ UpdateResourceRequest::decode_from(&request.to_bytes()).unwrap(),
+ request
+ );
+ }
+
+ fn reset_keys(names: &[&str]) -> WireResetKeys {
+ WireResetKeys::try_from(
+ names
+ .iter()
+ .map(|name| WireName::new(*name).unwrap())
+ .collect::<Vec<_>>(),
+ )
+ .unwrap()
+ }
+
+ fn options(entries: &[(u8, &[u8], u8, &[u8])]) -> WireOptions {
+ let mut buf = BytesMut::new();
+ encode_user_headers(entries, &mut buf);
+ WireOptions::from_bytes(buf.freeze()).unwrap()
+ }
+}
diff --git a/core/binary_protocol/src/requests/topics/mod.rs b/core/binary_protocol/src/requests/topics/mod.rs
index c9708e02d..69909986e 100644
--- a/core/binary_protocol/src/requests/topics/mod.rs
+++ b/core/binary_protocol/src/requests/topics/mod.rs
@@ -20,11 +20,11 @@ pub mod create_topic_with_assignments;
pub mod delete_topic;
pub mod get_topic;
pub mod get_topics;
-pub mod update_topic;
+pub mod update_topic_resolved;
pub use create_topic::CreateTopicRequest;
pub use create_topic_with_assignments::CreateTopicWithAssignmentsRequest;
pub use delete_topic::DeleteTopicRequest;
pub use get_topic::GetTopicRequest;
pub use get_topics::GetTopicsRequest;
-pub use update_topic::UpdateTopicRequest;
+pub use update_topic_resolved::UpdateTopicResolvedRequest;
diff --git a/core/binary_protocol/src/requests/topics/update_topic.rs b/core/binary_protocol/src/requests/topics/update_topic.rs
deleted file mode 100644
index 5f8865a3a..000000000
--- a/core/binary_protocol/src/requests/topics/update_topic.rs
+++ /dev/null
@@ -1,158 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-use crate::WireError;
-use crate::WireIdentifier;
-use crate::codec::{WireDecode, WireEncode};
-use crate::primitives::identifier::WireName;
-use crate::primitives::options::WireOptions;
-use bytes::BytesMut;
-
-/// `UpdateTopic` request.
-///
-/// Wire format:
-/// `[stream_id:WireIdentifier][topic_id:WireIdentifier][name_len:u8][name:N]
-/// [options TLV to end]`
-///
-/// The options block shares `CreateTopic`'s encoding, but the server validates
-/// which keys can be updated. Supporting a key at creation does not make it
-/// mutable. Absent keys are left unchanged, so updating the name does not erase
-/// settings introduced after the client was built.
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct UpdateTopicRequest {
- pub stream_id: WireIdentifier,
- pub topic_id: WireIdentifier,
- pub name: WireName,
- pub options: WireOptions,
-}
-
-impl WireEncode for UpdateTopicRequest {
- fn encoded_size(&self) -> usize {
- self.stream_id.encoded_size()
- + self.topic_id.encoded_size()
- + self.name.encoded_size()
- + self.options.encoded_size()
- }
-
- fn encode(&self, buf: &mut BytesMut) {
- self.stream_id.encode(buf);
- self.topic_id.encode(buf);
- self.name.encode(buf);
- self.options.encode(buf);
- }
-}
-
-impl WireDecode for UpdateTopicRequest {
- fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
- let (stream_id, mut pos) = WireIdentifier::decode(buf)?;
- let (topic_id, consumed) = WireIdentifier::decode(&buf[pos..])?;
- pos += consumed;
- let (name, name_consumed) = WireName::decode(&buf[pos..])?;
- pos += name_consumed;
- let options = WireOptions::from_slice(&buf[pos..])?;
- Ok((
- Self {
- stream_id,
- topic_id,
- name,
- options,
- },
- buf.len(),
- ))
- }
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
- use crate::primitives::user_headers::encode_user_headers;
-
- fn sample_options() -> WireOptions {
- let mut buf = BytesMut::new();
- encode_user_headers(&[(2, b"future_option", 9, &[3u8])], &mut buf);
- WireOptions::from_bytes(buf.freeze()).unwrap()
- }
-
- fn sample_request() -> UpdateTopicRequest {
- UpdateTopicRequest {
- stream_id: WireIdentifier::numeric(1),
- topic_id: WireIdentifier::numeric(2),
- name: WireName::new("updated-topic").unwrap(),
- options: sample_options(),
- }
- }
-
- #[test]
- fn roundtrip() {
- let req = sample_request();
- let bytes = req.to_bytes();
- let (decoded, consumed) = UpdateTopicRequest::decode(&bytes).unwrap();
- assert_eq!(consumed, bytes.len());
- assert_eq!(decoded, req);
- }
-
- #[test]
- fn roundtrip_named_identifiers() {
- let req = UpdateTopicRequest {
- stream_id: WireIdentifier::named("stream-a").unwrap(),
- topic_id: WireIdentifier::named("topic-b").unwrap(),
- name: WireName::new("new-name").unwrap(),
- options: WireOptions::empty(),
- };
- let bytes = req.to_bytes();
- let (decoded, consumed) = UpdateTopicRequest::decode(&bytes).unwrap();
- assert_eq!(consumed, bytes.len());
- assert_eq!(decoded, req);
- }
-
- #[test]
- fn truncated_options_return_error() {
- // One key-value pair, so any strict-interior truncation of the block
- // is structurally invalid. Cutting at the block boundary is NOT an
- // error: that is a legitimate update carrying no options.
- let req = sample_request();
- let bytes = req.to_bytes();
- let fixed_end = bytes.len() - req.options.encoded_size();
- for i in fixed_end + 1..bytes.len() {
- assert!(
- UpdateTopicRequest::decode(&bytes[..i]).is_err(),
- "expected error for truncation at byte {i}"
- );
- }
- }
-
- #[test]
- fn truncated_fixed_fields_return_error() {
- let req = UpdateTopicRequest {
- options: WireOptions::empty(),
- ..sample_request()
- };
- let bytes = req.to_bytes();
- for i in 0..bytes.len() {
- assert!(
- UpdateTopicRequest::decode(&bytes[..i]).is_err(),
- "expected error for truncation at byte {i}"
- );
- }
- }
-
- #[test]
- fn encoded_size_matches_output() {
- let req = sample_request();
- assert_eq!(req.encoded_size(), req.to_bytes().len());
- }
-}
diff --git a/core/binary_protocol/src/requests/topics/update_topic_resolved.rs b/core/binary_protocol/src/requests/topics/update_topic_resolved.rs
new file mode 100644
index 000000000..0fcfb52c5
--- /dev/null
+++ b/core/binary_protocol/src/requests/topics/update_topic_resolved.rs
@@ -0,0 +1,113 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::WireError;
+use crate::WireIdentifier;
+use crate::codec::{WireDecode, WireEncode};
+use crate::primitives::identifier::WireName;
+use crate::primitives::options::WireOptions;
+use crate::requests::system::update_resource::{
+ decode_optional_name, decode_resolved_blocks, encode_optional_name, encode_resolved_blocks,
+ optional_name_size, resolved_blocks_size,
+};
+use bytes::BytesMut;
+
+/// An `UpdateResource` on a topic after prepare: the rename extracted, the set
+/// keys canonical in `explicit_options`, the reset keys resolved to their
+/// defaults in `derived_options`.
+///
+/// Wire format:
+/// ```text
+/// [stream_id][topic_id][has_name:u8][name?][explicit_len:u32][explicit][derived_len:u32][derived]
+/// ```
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct UpdateTopicResolvedRequest {
+ pub stream_id: WireIdentifier,
+ pub topic_id: WireIdentifier,
+ pub name: Option<WireName>,
+ pub explicit_options: WireOptions,
+ pub derived_options: WireOptions,
+}
+
+impl WireEncode for UpdateTopicResolvedRequest {
+ fn encoded_size(&self) -> usize {
+ self.stream_id.encoded_size()
+ + self.topic_id.encoded_size()
+ + optional_name_size(self.name.as_ref())
+ + resolved_blocks_size(&self.explicit_options, &self.derived_options)
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ self.stream_id.encode(buf);
+ self.topic_id.encode(buf);
+ encode_optional_name(self.name.as_ref(), buf);
+ encode_resolved_blocks(&self.explicit_options, &self.derived_options, buf);
+ }
+}
+
+impl WireDecode for UpdateTopicResolvedRequest {
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let (stream_id, mut position) = WireIdentifier::decode(buf)?;
+ let (topic_id, consumed) = WireIdentifier::decode(&buf[position..])?;
+ position += consumed;
+ let (name, consumed) = decode_optional_name(buf, position)?;
+ position += consumed;
+ let (explicit_options, derived_options) = decode_resolved_blocks(buf, position)?;
+ Ok((
+ Self {
+ stream_id,
+ topic_id,
+ name,
+ explicit_options,
+ derived_options,
+ },
+ buf.len(),
+ ))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::primitives::user_headers::encode_user_headers;
+
+ #[test]
+ fn given_resolved_patch_when_roundtripping_should_preserve_columns_and_provenance() {
+ let request = UpdateTopicResolvedRequest {
+ stream_id: WireIdentifier::numeric(1),
+ topic_id: WireIdentifier::named("orders").unwrap(),
+ name: Some(WireName::new("orders-v2").unwrap()),
+ explicit_options: options(b"message_expiry", &7u64.to_le_bytes()),
+ derived_options: options(b"max_topic_size", &u64::MAX.to_le_bytes()),
+ };
+ let bytes = request.to_bytes();
+ assert_eq!(bytes.len(), request.encoded_size());
+ assert_eq!(
+ UpdateTopicResolvedRequest::decode_from(&bytes).unwrap(),
+ request
+ );
+ for length in 0..bytes.len() {
+ assert!(UpdateTopicResolvedRequest::decode(&bytes[..length]).is_err());
+ }
+ }
+
+ fn options(key: &[u8], value: &[u8]) -> WireOptions {
+ let mut buf = BytesMut::new();
+ encode_user_headers(&[(2, key, 12, value)], &mut buf);
+ WireOptions::from_bytes(buf.freeze()).unwrap()
+ }
+}
diff --git a/core/binary_protocol/src/requests/users/mod.rs b/core/binary_protocol/src/requests/users/mod.rs
index 59860b5eb..5e15e5fbd 100644
--- a/core/binary_protocol/src/requests/users/mod.rs
+++ b/core/binary_protocol/src/requests/users/mod.rs
@@ -25,7 +25,7 @@ pub mod login_register_with_pat;
pub mod login_user;
pub mod logout_user;
pub mod update_permissions;
-pub mod update_user;
+pub mod update_user_resolved;
pub use change_password::ChangePasswordRequest;
pub use create_user::CreateUserRequest;
@@ -37,4 +37,4 @@ pub use login_register_with_pat::LoginRegisterWithPatRequest;
pub use login_user::LoginUserRequest;
pub use logout_user::LogoutUserRequest;
pub use update_permissions::UpdatePermissionsRequest;
-pub use update_user::UpdateUserRequest;
+pub use update_user_resolved::UpdateUserResolvedRequest;
diff --git a/core/binary_protocol/src/requests/users/update_user.rs b/core/binary_protocol/src/requests/users/update_user.rs
deleted file mode 100644
index 125e1df02..000000000
--- a/core/binary_protocol/src/requests/users/update_user.rs
+++ /dev/null
@@ -1,228 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-use crate::WireError;
-use crate::WireIdentifier;
-use crate::codec::{WireDecode, WireEncode, read_u8};
-use crate::primitives::identifier::WireName;
-use crate::primitives::options::WireOptions;
-use bytes::{BufMut, BytesMut};
-
-/// `UpdateUser` request.
-///
-/// Wire format:
-/// `[user_id:WireIdentifier][has_username:u8][username_len:u8?][username:N?]
-/// [has_status:u8][status:u8?][options TLV to end]`
-///
-/// The options block mirrors `CreateUser`'s, so a user setting added to the
-/// catalog is updatable without another layout change. Keys absent from the
-/// block are left alone rather than reset: an update patches the stored map,
-/// so a client built before a key existed cannot erase it.
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct UpdateUserRequest {
- pub user_id: WireIdentifier,
- pub username: Option<WireName>,
- pub status: Option<u8>,
- pub options: WireOptions,
-}
-
-impl WireEncode for UpdateUserRequest {
- fn encoded_size(&self) -> usize {
- self.user_id.encoded_size()
- + 1 // has_username
- + self.username.as_ref().map_or(0, WireEncode::encoded_size)
- + 1 // has_status
- + self.status.map_or(0, |_| 1)
- + self.options.encoded_size()
- }
-
- fn encode(&self, buf: &mut BytesMut) {
- self.user_id.encode(buf);
- match &self.username {
- Some(name) => {
- buf.put_u8(1);
- name.encode(buf);
- }
- None => {
- buf.put_u8(0);
- }
- }
- match self.status {
- Some(s) => {
- buf.put_u8(1);
- buf.put_u8(s);
- }
- None => {
- buf.put_u8(0);
- }
- }
- self.options.encode(buf);
- }
-}
-
-impl WireDecode for UpdateUserRequest {
- fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
- let (user_id, mut pos) = WireIdentifier::decode(buf)?;
-
- let has_username = read_u8(buf, pos)?;
- pos += 1;
- let username = if has_username == 1 {
- let (name, consumed) = WireName::decode(&buf[pos..])?;
- pos += consumed;
- Some(name)
- } else {
- None
- };
-
- let has_status = read_u8(buf, pos)?;
- pos += 1;
- let status = if has_status == 1 {
- let s = read_u8(buf, pos)?;
- pos += 1;
- Some(s)
- } else {
- None
- };
-
- let options = WireOptions::from_slice(&buf[pos..])?;
- Ok((
- Self {
- user_id,
- username,
- status,
- options,
- },
- buf.len(),
- ))
- }
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
- use crate::primitives::user_headers::encode_user_headers;
-
- fn sample_options() -> WireOptions {
- let mut buf = BytesMut::new();
- encode_user_headers(&[(2, b"future_key", 2, b"value")], &mut buf);
- WireOptions::from_bytes(buf.freeze()).unwrap()
- }
-
- #[test]
- fn roundtrip_both_present() {
- let req = UpdateUserRequest {
- user_id: WireIdentifier::numeric(1),
- username: Some(WireName::new("new-name").unwrap()),
- status: Some(2),
- options: WireOptions::empty(),
- };
- let bytes = req.to_bytes();
- let (decoded, consumed) = UpdateUserRequest::decode(&bytes).unwrap();
- assert_eq!(consumed, bytes.len());
- assert_eq!(decoded, req);
- }
-
- #[test]
- fn roundtrip_both_none() {
- let req = UpdateUserRequest {
- user_id: WireIdentifier::numeric(5),
- username: None,
- status: None,
- options: WireOptions::empty(),
- };
- let bytes = req.to_bytes();
- let (decoded, consumed) = UpdateUserRequest::decode(&bytes).unwrap();
- assert_eq!(consumed, bytes.len());
- assert_eq!(decoded, req);
- }
-
- #[test]
- fn roundtrip_username_only() {
- let req = UpdateUserRequest {
- user_id: WireIdentifier::named("admin").unwrap(),
- username: Some(WireName::new("super-admin").unwrap()),
- status: None,
- options: WireOptions::empty(),
- };
- let bytes = req.to_bytes();
- let (decoded, consumed) = UpdateUserRequest::decode(&bytes).unwrap();
- assert_eq!(consumed, bytes.len());
- assert_eq!(decoded, req);
- }
-
- #[test]
- fn roundtrip_status_only() {
- let req = UpdateUserRequest {
- user_id: WireIdentifier::numeric(10),
- username: None,
- status: Some(0),
- options: WireOptions::empty(),
- };
- let bytes = req.to_bytes();
- let (decoded, consumed) = UpdateUserRequest::decode(&bytes).unwrap();
- assert_eq!(consumed, bytes.len());
- assert_eq!(decoded, req);
- }
-
- #[test]
- fn encoded_size_matches_output() {
- let req = UpdateUserRequest {
- user_id: WireIdentifier::numeric(1),
- username: Some(WireName::new("test").unwrap()),
- status: Some(1),
- options: WireOptions::empty(),
- };
- assert_eq!(req.encoded_size(), req.to_bytes().len());
- }
-
- #[test]
- fn truncated_options_return_error() {
- // One pair, so any strict-interior truncation is invalid. Cutting at
- // the block boundary is a legitimate update carrying no options.
- let req = UpdateUserRequest {
- user_id: WireIdentifier::numeric(1),
- username: Some(WireName::new("name").unwrap()),
- status: Some(1),
- options: sample_options(),
- };
- let bytes = req.to_bytes();
- let fixed_end = bytes.len() - req.options.encoded_size();
- for i in fixed_end + 1..bytes.len() {
- assert!(
- UpdateUserRequest::decode(&bytes[..i]).is_err(),
- "expected error for truncation at byte {i}"
- );
- }
- }
-
- #[test]
- fn truncated_returns_error() {
- let req = UpdateUserRequest {
- user_id: WireIdentifier::numeric(1),
- username: Some(WireName::new("name").unwrap()),
- status: Some(1),
- options: WireOptions::empty(),
- };
- let bytes = req.to_bytes();
- for i in 0..bytes.len() {
- assert!(
- UpdateUserRequest::decode(&bytes[..i]).is_err(),
- "expected error for truncation at byte {i}"
- );
- }
- }
-}
diff --git a/core/binary_protocol/src/requests/users/update_user_resolved.rs b/core/binary_protocol/src/requests/users/update_user_resolved.rs
new file mode 100644
index 000000000..1a009d73d
--- /dev/null
+++ b/core/binary_protocol/src/requests/users/update_user_resolved.rs
@@ -0,0 +1,136 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::WireError;
+use crate::WireIdentifier;
+use crate::codec::{WireDecode, WireEncode, read_u8};
+use crate::primitives::identifier::WireName;
+use crate::primitives::options::WireOptions;
+use crate::requests::system::update_resource::{
+ decode_optional_name, decode_resolved_blocks, encode_optional_name, encode_resolved_blocks,
+ optional_name_size, resolved_blocks_size,
+};
+use bytes::{BufMut, BytesMut};
+use std::borrow::Cow;
+
+/// An `UpdateResource` on a user after prepare: the username and status
+/// extracted, the set keys canonical in `explicit_options`, the reset keys
+/// resolved to their defaults in `derived_options`.
+///
+/// Wire format:
+/// ```text
+/// [user_id][has_username:u8][username?][has_status:u8][status:u8?]
+/// [explicit_len:u32][explicit][derived_len:u32][derived]
+/// ```
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct UpdateUserResolvedRequest {
+ pub user_id: WireIdentifier,
+ pub username: Option<WireName>,
+ pub status: Option<u8>,
+ pub explicit_options: WireOptions,
+ pub derived_options: WireOptions,
+}
+
+impl WireEncode for UpdateUserResolvedRequest {
+ fn encoded_size(&self) -> usize {
+ self.user_id.encoded_size()
+ + optional_name_size(self.username.as_ref())
+ + 1
+ + usize::from(self.status.is_some())
+ + resolved_blocks_size(&self.explicit_options, &self.derived_options)
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ self.user_id.encode(buf);
+ encode_optional_name(self.username.as_ref(), buf);
+ match self.status {
+ Some(status) => {
+ buf.put_u8(1);
+ buf.put_u8(status);
+ }
+ None => buf.put_u8(0),
+ }
+ encode_resolved_blocks(&self.explicit_options, &self.derived_options, buf);
+ }
+}
+
+impl WireDecode for UpdateUserResolvedRequest {
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let (user_id, mut position) = WireIdentifier::decode(buf)?;
+ let (username, consumed) = decode_optional_name(buf, position)?;
+ position += consumed;
+ let status = match read_u8(buf, position)? {
+ 0 => None,
+ 1 => {
+ position += 1;
+ Some(read_u8(buf, position)?)
+ }
+ _ => {
+ return Err(WireError::Validation(Cow::Borrowed(
+ "presence byte must be 0 or 1",
+ )));
+ }
+ };
+ position += 1;
+ let (explicit_options, derived_options) = decode_resolved_blocks(buf, position)?;
+ Ok((
+ Self {
+ user_id,
+ username,
+ status,
+ explicit_options,
+ derived_options,
+ },
+ buf.len(),
+ ))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn given_resolved_patch_when_roundtripping_should_preserve_columns_and_provenance() {
+ for (username, status) in [
+ (None, None),
+ (Some(WireName::new("alice2").unwrap()), None),
+ (None, Some(2)),
+ (Some(WireName::new("alice2").unwrap()), Some(1)),
+ ] {
+ let request = UpdateUserResolvedRequest {
+ user_id: WireIdentifier::named("alice").unwrap(),
+ username,
+ status,
+ explicit_options: WireOptions::empty(),
+ derived_options: WireOptions::empty(),
+ };
+ let bytes = request.to_bytes();
+ assert_eq!(bytes.len(), request.encoded_size());
+ assert_eq!(
+ UpdateUserResolvedRequest::decode_from(&bytes).unwrap(),
+ request
+ );
+ for length in 0..bytes.len() {
+ assert!(UpdateUserResolvedRequest::decode(&bytes[..length]).is_err());
+ }
+ }
+ let mut bad_status = WireIdentifier::numeric(1).to_bytes().to_vec();
+ bad_status.extend_from_slice(&[0, 3]);
+ assert!(UpdateUserResolvedRequest::decode(&bad_status).is_err());
+ }
+}
diff --git a/core/binary_protocol/src/responses/streams/get_stream.rs b/core/binary_protocol/src/responses/streams/get_stream.rs
index fd5b2acdc..b8b6aa3e7 100644
--- a/core/binary_protocol/src/responses/streams/get_stream.rs
+++ b/core/binary_protocol/src/responses/streams/get_stream.rs
@@ -16,7 +16,7 @@
// under the License.
use crate::WireError;
-use crate::codec::{WireDecode, WireEncode, read_u8, read_u32_le, read_u64_le};
+use crate::codec::{WireDecode, WireEncode, read_u32_le, read_u64_le};
use crate::primitives::identifier::WireName;
use crate::primitives::options::{
WireOptions, decode_options_prefixed, encode_options_prefixed, options_prefixed_size,
@@ -26,10 +26,9 @@ use bytes::{BufMut, BytesMut};
/// Topic header within a `GetStream` response.
///
-/// Wire format (50 + `name_len` + options bytes):
+/// Wire format (33 + `name_len` + options bytes):
/// ```text
-/// [id:4][created_at:8][partitions_count:4][message_expiry:8]
-/// [compression_algorithm:1][max_topic_size:8]
+/// [id:4][created_at:8][partitions_count:4]
/// [size_bytes:8][messages_count:8][name_len:1][name:N]
/// [explicit_options_len:4][explicit options TLV]
/// [derived_options_len:4][derived options TLV]
@@ -38,14 +37,12 @@ use bytes::{BufMut, BytesMut};
/// `options` are the keys the client pinned at create; `derived_options` are
/// the defaults the admitting primary resolved. A round-tripping client
/// re-sends only `options`; the effective configuration is the union.
+/// Message expiry, compression and the size cap live in those blocks only.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TopicHeader {
pub id: u32,
pub created_at: u64,
pub partitions_count: u32,
- pub message_expiry: u64,
- pub compression_algorithm: u8,
- pub max_topic_size: u64,
pub size_bytes: u64,
pub messages_count: u64,
pub name: WireName,
@@ -54,7 +51,7 @@ pub struct TopicHeader {
}
impl TopicHeader {
- pub(crate) const FIXED_SIZE: usize = 4 + 8 + 4 + 8 + 1 + 8 + 8 + 8 + 1; // 50
+ pub(crate) const FIXED_SIZE: usize = 4 + 8 + 4 + 8 + 8 + 1; // 33
}
impl WireEncode for TopicHeader {
@@ -69,9 +66,6 @@ impl WireEncode for TopicHeader {
buf.put_u32_le(self.id);
buf.put_u64_le(self.created_at);
buf.put_u32_le(self.partitions_count);
- buf.put_u64_le(self.message_expiry);
- buf.put_u8(self.compression_algorithm);
- buf.put_u64_le(self.max_topic_size);
buf.put_u64_le(self.size_bytes);
buf.put_u64_le(self.messages_count);
self.name.encode(buf);
@@ -85,13 +79,10 @@ impl WireDecode for TopicHeader {
let id = read_u32_le(buf, 0)?;
let created_at = read_u64_le(buf, 4)?;
let partitions_count = read_u32_le(buf, 12)?;
- let message_expiry = read_u64_le(buf, 16)?;
- let compression_algorithm = read_u8(buf, 24)?;
- let max_topic_size = read_u64_le(buf, 25)?;
- let size_bytes = read_u64_le(buf, 33)?;
- let messages_count = read_u64_le(buf, 41)?;
- let (name, name_consumed) = WireName::decode(&buf[49..])?;
- let mut consumed = 49 + name_consumed;
+ let size_bytes = read_u64_le(buf, 16)?;
+ let messages_count = read_u64_le(buf, 24)?;
+ let (name, name_consumed) = WireName::decode(&buf[32..])?;
+ let mut consumed = 32 + name_consumed;
let (options, options_consumed) = decode_options_prefixed(buf, consumed)?;
consumed += options_consumed;
let (derived_options, derived_consumed) = decode_options_prefixed(buf, consumed)?;
@@ -102,9 +93,6 @@ impl WireDecode for TopicHeader {
id,
created_at,
partitions_count,
- message_expiry,
- compression_algorithm,
- max_topic_size,
size_bytes,
messages_count,
name,
@@ -186,6 +174,7 @@ mod tests {
messages_count: 200,
name: WireName::new("my-stream").unwrap(),
options: WireOptions::empty(),
+ derived_options: WireOptions::empty(),
}
}
@@ -194,9 +183,6 @@ mod tests {
id,
created_at: 1_710_000_000_000,
partitions_count: 3,
- message_expiry: 0,
- compression_algorithm: 1,
- max_topic_size: 0,
size_bytes: 1024,
messages_count: 100,
name: WireName::new(name).unwrap(),
diff --git a/core/binary_protocol/src/responses/streams/get_streams.rs b/core/binary_protocol/src/responses/streams/get_streams.rs
index 11e30ab55..33d3acdc5 100644
--- a/core/binary_protocol/src/responses/streams/get_streams.rs
+++ b/core/binary_protocol/src/responses/streams/get_streams.rs
@@ -85,6 +85,7 @@ mod tests {
messages_count: 50,
name: WireName::new("s1").unwrap(),
options: WireOptions::empty(),
+ derived_options: WireOptions::empty(),
},
StreamResponse {
id: 2,
@@ -94,6 +95,7 @@ mod tests {
messages_count: 0,
name: WireName::new("stream-two").unwrap(),
options: WireOptions::empty(),
+ derived_options: WireOptions::empty(),
},
],
};
diff --git a/core/binary_protocol/src/responses/streams/mod.rs b/core/binary_protocol/src/responses/streams/mod.rs
index 7b869d29f..09fe7ec03 100644
--- a/core/binary_protocol/src/responses/streams/mod.rs
+++ b/core/binary_protocol/src/responses/streams/mod.rs
@@ -20,7 +20,6 @@ mod delete_stream;
pub mod get_stream;
pub mod get_streams;
mod stream_response;
-mod update_stream;
pub use super::EmptyResponse;
pub use create_stream::CreateStreamResponse;
@@ -28,4 +27,3 @@ pub use delete_stream::DeleteStreamResponse;
pub use get_stream::{GetStreamResponse, TopicHeader};
pub use get_streams::GetStreamsResponse;
pub use stream_response::StreamResponse;
-pub use update_stream::UpdateStreamResponse;
diff --git a/core/binary_protocol/src/responses/streams/stream_response.rs b/core/binary_protocol/src/responses/streams/stream_response.rs
index 8beb76f6d..7253d20d8 100644
--- a/core/binary_protocol/src/responses/streams/stream_response.rs
+++ b/core/binary_protocol/src/responses/streams/stream_response.rs
@@ -28,11 +28,12 @@ use bytes::{BufMut, BytesMut};
/// Wire format (33 + `name_len` + options bytes):
/// ```text
/// [id:4][created_at:8][topics_count:4][size_bytes:8][messages_count:8][name_len:1][name:N]
-/// [options_len:4][options TLV]
+/// [explicit_options_len:4][explicit options TLV]
+/// [derived_options_len:4][derived options TLV]
/// ```
///
-/// Streams have no server-derived options (no catalog keys resolve against
-/// config yet), so a single client-explicit block suffices.
+/// `options` are the keys the client set, `derived_options` the values the
+/// server filled or computes, such as the encryption key source.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct StreamResponse {
pub id: u32,
@@ -42,6 +43,7 @@ pub struct StreamResponse {
pub messages_count: u64,
pub name: WireName,
pub options: WireOptions,
+ pub derived_options: WireOptions,
}
impl StreamResponse {
@@ -50,7 +52,10 @@ impl StreamResponse {
impl WireEncode for StreamResponse {
fn encoded_size(&self) -> usize {
- Self::FIXED_SIZE + self.name.len() + options_prefixed_size(&self.options)
+ Self::FIXED_SIZE
+ + self.name.len()
+ + options_prefixed_size(&self.options)
+ + options_prefixed_size(&self.derived_options)
}
fn encode(&self, buf: &mut BytesMut) {
@@ -61,6 +66,7 @@ impl WireEncode for StreamResponse {
buf.put_u64_le(self.messages_count);
self.name.encode(buf);
encode_options_prefixed(&self.options, buf);
+ encode_options_prefixed(&self.derived_options, buf);
}
}
@@ -75,6 +81,8 @@ impl WireDecode for StreamResponse {
let mut consumed = 32 + name_consumed;
let (options, options_consumed) = decode_options_prefixed(buf, consumed)?;
consumed += options_consumed;
+ let (derived_options, derived_consumed) = decode_options_prefixed(buf, consumed)?;
+ consumed += derived_consumed;
Ok((
Self {
@@ -85,6 +93,7 @@ impl WireDecode for StreamResponse {
messages_count,
name,
options,
+ derived_options,
},
consumed,
))
@@ -104,6 +113,7 @@ mod tests {
messages_count: 100,
name: WireName::new("test-stream").unwrap(),
options: WireOptions::empty(),
+ derived_options: WireOptions::empty(),
}
}
@@ -111,7 +121,7 @@ mod tests {
fn roundtrip() {
let resp = sample();
let bytes = resp.to_bytes();
- assert_eq!(bytes.len(), StreamResponse::FIXED_SIZE + 11 + 4);
+ assert_eq!(bytes.len(), StreamResponse::FIXED_SIZE + 11 + 4 + 4);
let (decoded, consumed) = StreamResponse::decode(&bytes).unwrap();
assert_eq!(consumed, bytes.len());
assert_eq!(decoded, resp);
diff --git a/core/binary_protocol/src/responses/streams/update_stream.rs b/core/binary_protocol/src/responses/streams/update_stream.rs
deleted file mode 100644
index b002d9c86..000000000
--- a/core/binary_protocol/src/responses/streams/update_stream.rs
+++ /dev/null
@@ -1,19 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-/// `UpdateStream` response is empty.
-pub type UpdateStreamResponse = super::EmptyResponse;
diff --git a/core/binary_protocol/src/responses/system/describe_options.rs b/core/binary_protocol/src/responses/system/describe_options.rs
index 154196ef7..3b12d7629 100644
--- a/core/binary_protocol/src/responses/system/describe_options.rs
+++ b/core/binary_protocol/src/responses/system/describe_options.rs
@@ -24,33 +24,36 @@ use bytes::{BufMut, Bytes, BytesMut};
///
/// Wire format:
/// ```text
-/// [key_len:u8][key:N][kind:u8]
+/// [key_len:u8][key:N][kind:u8][flags:u8]
/// [default_len:u32][default bytes][description_len:u32][description]
/// ```
///
/// `kind` is this key's canonical header kind code: what the server encodes its
-/// default under, and what a value set at create is stored as, since create
-/// admission re-encodes the block from its own parse (a `String` value parsed by
-/// the same rules as a config file value is accepted and canonicalized). An
-/// update stores the client's bytes verbatim and is the exception. `default` is
-/// that default in `kind`'s encoding, empty when the key has none, and is a
-/// build constant rather than a per-node value.
+/// default under, and what a value set at create or update is stored as, since
+/// admission re-encodes every block from its own parse (a `String` value parsed
+/// by the same rules as a config file value is accepted and canonicalized).
+/// `flags` bits: 0 updatable, 1 resettable, 2 secret (write-only), 3 read-only
+/// (server-computed), 4 critical. Higher bits are zero and readers ignore them.
+/// `default` is that default in `kind`'s encoding, empty when the key has none,
+/// and is a build constant rather than a per-node value.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OptionDescriptor {
pub key: WireName,
pub kind: u8,
+ pub flags: u8,
pub default_value: Bytes,
pub description: String,
}
impl WireEncode for OptionDescriptor {
fn encoded_size(&self) -> usize {
- self.key.encoded_size() + 1 + 4 + self.default_value.len() + 4 + self.description.len()
+ self.key.encoded_size() + 1 + 1 + 4 + self.default_value.len() + 4 + self.description.len()
}
fn encode(&self, buf: &mut BytesMut) {
self.key.encode(buf);
buf.put_u8(self.kind);
+ buf.put_u8(self.flags);
#[allow(clippy::cast_possible_truncation)]
buf.put_u32_le(self.default_value.len() as u32);
buf.put_slice(&self.default_value);
@@ -65,6 +68,8 @@ impl WireDecode for OptionDescriptor {
let (key, mut pos) = WireName::decode(buf)?;
let kind = read_u8(buf, pos)?;
pos += 1;
+ let flags = read_u8(buf, pos)?;
+ pos += 1;
let default_len = read_u32_le(buf, pos)? as usize;
pos += 4;
// `checked_add`: on a 32-bit target a wire-supplied length wraps, the
@@ -108,6 +113,7 @@ impl WireDecode for OptionDescriptor {
Self {
key,
kind,
+ flags,
default_value,
description,
},
@@ -143,9 +149,9 @@ impl WireEncode for DescribeOptionsResponse {
}
/// Smallest a descriptor can encode as: a one-character `WireName` (a length
-/// byte plus at least one byte of name), a kind byte, and empty
+/// byte plus at least one byte of name), a kind byte, a flags byte, and empty
/// length-prefixed default and description.
-const MIN_DESCRIPTOR_SIZE: usize = 2 + 1 + 4 + 4;
+const MIN_DESCRIPTOR_SIZE: usize = 2 + 1 + 1 + 4 + 4;
impl WireDecode for DescribeOptionsResponse {
fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
@@ -175,12 +181,14 @@ mod tests {
OptionDescriptor {
key: WireName::new("partitions_count").unwrap(),
kind: 11,
+ flags: 0,
default_value: Bytes::copy_from_slice(&1u32.to_le_bytes()),
description: "Number of partitions to create".to_string(),
},
OptionDescriptor {
key: WireName::new("max_topic_size").unwrap(),
kind: 12,
+ flags: 0b11,
default_value: Bytes::new(),
description: String::new(),
},
diff --git a/core/binary_protocol/src/responses/system/mod.rs b/core/binary_protocol/src/responses/system/mod.rs
index 113eee391..a8e21d3b1 100644
--- a/core/binary_protocol/src/responses/system/mod.rs
+++ b/core/binary_protocol/src/responses/system/mod.rs
@@ -21,6 +21,7 @@ pub mod get_me;
pub mod get_snapshot;
pub mod get_stats;
mod ping;
+mod update_resource;
pub use super::EmptyResponse;
pub use describe_options::{DescribeOptionsResponse, OptionDescriptor};
@@ -29,3 +30,4 @@ pub use get_me::GetMeResponse;
pub use get_snapshot::GetSnapshotResponse;
pub use get_stats::StatsResponse;
pub use ping::PingResponse;
+pub use update_resource::UpdateResourceResponse;
diff --git a/core/binary_protocol/src/responses/topics/update_topic.rs b/core/binary_protocol/src/responses/system/update_resource.rs
similarity index 89%
rename from core/binary_protocol/src/responses/topics/update_topic.rs
rename to core/binary_protocol/src/responses/system/update_resource.rs
index 6df77c408..3645eb53d 100644
--- a/core/binary_protocol/src/responses/topics/update_topic.rs
+++ b/core/binary_protocol/src/responses/system/update_resource.rs
@@ -15,5 +15,5 @@
// specific language governing permissions and limitations
// under the License.
-/// `UpdateTopic` response is empty.
-pub type UpdateTopicResponse = super::EmptyResponse;
+/// `UpdateResource` response is empty.
+pub type UpdateResourceResponse = super::EmptyResponse;
diff --git a/core/binary_protocol/src/responses/topics/get_topic.rs b/core/binary_protocol/src/responses/topics/get_topic.rs
index 76d86727d..96cd8e80f 100644
--- a/core/binary_protocol/src/responses/topics/get_topic.rs
+++ b/core/binary_protocol/src/responses/topics/get_topic.rs
@@ -147,9 +147,6 @@ mod tests {
id: 1,
created_at: 1_710_000_000_000,
partitions_count,
- message_expiry: 0,
- compression_algorithm: 1,
- max_topic_size: 0,
size_bytes: 2048,
messages_count: 200,
name: WireName::new("my-topic").unwrap(),
diff --git a/core/binary_protocol/src/responses/topics/get_topics.rs b/core/binary_protocol/src/responses/topics/get_topics.rs
index 6852866e3..d8f5d9d44 100644
--- a/core/binary_protocol/src/responses/topics/get_topics.rs
+++ b/core/binary_protocol/src/responses/topics/get_topics.rs
@@ -89,9 +89,6 @@ mod tests {
id,
created_at: 1_710_000_000_000,
partitions_count: 3,
- message_expiry: 0,
- compression_algorithm: 1,
- max_topic_size: 0,
size_bytes: 1024,
messages_count: 100,
name: WireName::new(name).unwrap(),
diff --git a/core/binary_protocol/src/responses/topics/mod.rs b/core/binary_protocol/src/responses/topics/mod.rs
index a6567d196..f1279ee0d 100644
--- a/core/binary_protocol/src/responses/topics/mod.rs
+++ b/core/binary_protocol/src/responses/topics/mod.rs
@@ -19,11 +19,9 @@ mod create_topic;
mod delete_topic;
pub mod get_topic;
pub mod get_topics;
-mod update_topic;
pub use super::EmptyResponse;
pub use create_topic::CreateTopicResponse;
pub use delete_topic::DeleteTopicResponse;
pub use get_topic::{GetTopicResponse, PartitionResponse};
pub use get_topics::GetTopicsResponse;
-pub use update_topic::UpdateTopicResponse;
diff --git a/core/binary_protocol/src/responses/users/get_user.rs b/core/binary_protocol/src/responses/users/get_user.rs
index 6392e37ce..3628d23d6 100644
--- a/core/binary_protocol/src/responses/users/get_user.rs
+++ b/core/binary_protocol/src/responses/users/get_user.rs
@@ -101,6 +101,7 @@ mod tests {
status: 1,
username: WireName::new("admin").unwrap(),
options: WireOptions::empty(),
+ derived_options: WireOptions::empty(),
}
}
diff --git a/core/binary_protocol/src/responses/users/get_users.rs b/core/binary_protocol/src/responses/users/get_users.rs
index 87811f6f4..556f9edcd 100644
--- a/core/binary_protocol/src/responses/users/get_users.rs
+++ b/core/binary_protocol/src/responses/users/get_users.rs
@@ -81,6 +81,7 @@ mod tests {
status: 1,
username: WireName::new("admin").unwrap(),
options: WireOptions::empty(),
+ derived_options: WireOptions::empty(),
},
UserResponse {
id: 2,
@@ -88,6 +89,7 @@ mod tests {
status: 2,
username: WireName::new("alice").unwrap(),
options: WireOptions::empty(),
+ derived_options: WireOptions::empty(),
},
],
};
diff --git a/core/binary_protocol/src/responses/users/mod.rs b/core/binary_protocol/src/responses/users/mod.rs
index cb95cdd71..e7a85659b 100644
--- a/core/binary_protocol/src/responses/users/mod.rs
+++ b/core/binary_protocol/src/responses/users/mod.rs
@@ -24,7 +24,6 @@ pub mod login_register;
pub mod login_user;
mod logout_user;
mod update_permissions;
-mod update_user;
pub mod user_response;
pub use super::EmptyResponse;
@@ -37,5 +36,4 @@ pub use login_register::LoginRegisterResponse;
pub use login_user::IdentityResponse;
pub use logout_user::LogoutUserResponse;
pub use update_permissions::UpdatePermissionsResponse;
-pub use update_user::UpdateUserResponse;
pub use user_response::UserResponse;
diff --git a/core/binary_protocol/src/responses/users/update_user.rs b/core/binary_protocol/src/responses/users/update_user.rs
deleted file mode 100644
index 2b4a4e4be..000000000
--- a/core/binary_protocol/src/responses/users/update_user.rs
+++ /dev/null
@@ -1,18 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-pub type UpdateUserResponse = super::EmptyResponse;
diff --git a/core/binary_protocol/src/responses/users/user_response.rs b/core/binary_protocol/src/responses/users/user_response.rs
index ef495a9eb..9b777747c 100644
--- a/core/binary_protocol/src/responses/users/user_response.rs
+++ b/core/binary_protocol/src/responses/users/user_response.rs
@@ -28,11 +28,12 @@ use bytes::{BufMut, BytesMut};
/// Wire format (13 + `username_len` + options bytes):
/// ```text
/// [id:4][created_at:8][status:1][username_len:1][username:N]
-/// [options_len:4][options TLV]
+/// [explicit_options_len:4][explicit options TLV]
+/// [derived_options_len:4][derived options TLV]
/// ```
///
-/// Users have no server-derived options (no catalog keys resolve against
-/// config yet), so a single client-explicit block suffices.
+/// `options` are the keys the client set, `derived_options` the values the
+/// server filled.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct UserResponse {
pub id: u32,
@@ -40,6 +41,7 @@ pub struct UserResponse {
pub status: u8,
pub username: WireName,
pub options: WireOptions,
+ pub derived_options: WireOptions,
}
impl UserResponse {
@@ -48,7 +50,10 @@ impl UserResponse {
impl WireEncode for UserResponse {
fn encoded_size(&self) -> usize {
- Self::FIXED_SIZE + self.username.encoded_size() + options_prefixed_size(&self.options)
+ Self::FIXED_SIZE
+ + self.username.encoded_size()
+ + options_prefixed_size(&self.options)
+ + options_prefixed_size(&self.derived_options)
}
fn encode(&self, buf: &mut BytesMut) {
@@ -57,6 +62,7 @@ impl WireEncode for UserResponse {
buf.put_u8(self.status);
self.username.encode(buf);
encode_options_prefixed(&self.options, buf);
+ encode_options_prefixed(&self.derived_options, buf);
}
}
@@ -69,6 +75,8 @@ impl WireDecode for UserResponse {
let mut consumed = 13 + name_consumed;
let (options, options_consumed) = decode_options_prefixed(buf, consumed)?;
consumed += options_consumed;
+ let (derived_options, derived_consumed) = decode_options_prefixed(buf, consumed)?;
+ consumed += derived_consumed;
Ok((
Self {
@@ -77,6 +85,7 @@ impl WireDecode for UserResponse {
status,
username,
options,
+ derived_options,
},
consumed,
))
@@ -94,6 +103,7 @@ mod tests {
status: 1,
username: WireName::new("admin").unwrap(),
options: WireOptions::empty(),
+ derived_options: WireOptions::empty(),
}
}
@@ -101,7 +111,7 @@ mod tests {
fn roundtrip() {
let resp = sample();
let bytes = resp.to_bytes();
- assert_eq!(bytes.len(), UserResponse::FIXED_SIZE + 1 + 5 + 4);
+ assert_eq!(bytes.len(), UserResponse::FIXED_SIZE + 1 + 5 + 4 + 4);
let (decoded, consumed) = UserResponse::decode(&bytes).unwrap();
assert_eq!(consumed, bytes.len());
assert_eq!(decoded, resp);
diff --git a/core/binary_protocol/src/version.rs b/core/binary_protocol/src/version.rs
index adf1ba597..9dbe03838 100644
--- a/core/binary_protocol/src/version.rs
+++ b/core/binary_protocol/src/version.rs
@@ -102,17 +102,25 @@ const PATCH_MASK: u32 = COMPONENT_MAX;
/// Current development protocol version, independent of package versions.
///
/// Version 0.11.1 requires the session-binding secret in login requests.
+/// Version 0.12.0 replaces the per-resource update commands with the generic
+/// `UpdateResource` patch, drops the message expiry, compression and size cap
+/// fields from the topic header (50 to 33 fixed bytes, the values live in the
+/// option blocks), adds derived option blocks to stream and user responses and
+/// a flags byte to each `DescribeOptions` descriptor.
/// Finalize this number and the minimum during stable release preparation.
/// Released servers using protocol 0.11.0 accept every 0.11.x patch, so an
/// incompatible release needs a new minor version for those servers to reject it.
-pub const IGGY_PROTOCOL_VERSION: u32 = pack_protocol_version(0, 11, 1);
+pub const IGGY_PROTOCOL_VERSION: u32 = pack_protocol_version(0, 12, 0);
/// Oldest protocol version this build accepts at login.
-/// Version 0.11.0 has no session-binding secret and is incompatible.
+///
+/// Version 0.11.0 has no session-binding secret, and 0.11.1 still sends the
+/// per-resource update commands, decodes the 50-byte topic header and reads
+/// single-block stream and user responses, so both are incompatible.
///
/// Keep this equal to the current protocol unless compatibility with older
/// released protocols has been verified. Decide the range per stable release.
-pub const IGGY_PROTOCOL_VERSION_MIN: u32 = pack_protocol_version(0, 11, 1);
+pub const IGGY_PROTOCOL_VERSION_MIN: u32 = pack_protocol_version(0, 12, 0);
const _: () =
assert!(IGGY_PROTOCOL_VERSION_MIN > 0 && IGGY_PROTOCOL_VERSION_MIN <= IGGY_PROTOCOL_VERSION);
@@ -255,11 +263,11 @@ mod tests {
}
#[test]
- fn login_without_binding_secret_is_outside_the_compatible_window() {
+ fn login_from_a_per_resource_update_protocol_is_outside_the_compatible_window() {
assert!(!is_protocol_compatible(pack_protocol_version(0, 11, 0)));
- assert!(is_protocol_compatible(pack_protocol_version(0, 11, 1)));
- assert!(!is_protocol_compatible(pack_protocol_version(0, 11, 1023)));
- assert!(!is_protocol_compatible(pack_protocol_version(0, 12, 0)));
+ assert!(!is_protocol_compatible(pack_protocol_version(0, 11, 1)));
+ assert!(is_protocol_compatible(pack_protocol_version(0, 12, 0)));
+ assert!(!is_protocol_compatible(pack_protocol_version(0, 12, 1)));
}
#[test]
@@ -269,8 +277,8 @@ mod tests {
}
#[test]
- fn min_requires_the_session_binding_wire_layout() {
- assert_eq!(IGGY_PROTOCOL_VERSION_MIN, pack_protocol_version(0, 11, 1));
+ fn min_requires_the_generic_update_wire_layout() {
+ assert_eq!(IGGY_PROTOCOL_VERSION_MIN, pack_protocol_version(0, 12, 0));
}
#[test]
diff --git a/core/cli/Cargo.toml b/core/cli/Cargo.toml
index 5704e5421..f8d433742 100644
--- a/core/cli/Cargo.toml
+++ b/core/cli/Cargo.toml
@@ -17,7 +17,7 @@
[package]
name = "iggy-cli"
-version = "0.14.1-edge.1"
+version = "0.14.1-edge.2"
edition = "2024"
rust-version.workspace = true
authors = ["bartosz.ciesla@gmail.com"]
diff --git a/core/cli/src/args/common.rs b/core/cli/src/args/common.rs
index 2accf0117..08ae9940e 100644
--- a/core/cli/src/args/common.rs
+++ b/core/cli/src/args/common.rs
@@ -16,6 +16,7 @@
// under the License.
use clap::ValueEnum;
+use iggy::prelude::OptionKey;
use iggy_cli::commands::binary_client::get_clients::GetClientsOutput;
use iggy_cli::commands::binary_cluster::get_cluster_metadata::GetClusterMetadataOutput;
use iggy_cli::commands::binary_consumer_groups::get_consumer_groups::GetConsumerGroupsOutput;
@@ -25,6 +26,7 @@ use iggy_cli::commands::binary_streams::get_streams::GetStreamsOutput;
use iggy_cli::commands::binary_system::stats::GetStatsOutput;
use iggy_cli::commands::binary_topics::get_topics::GetTopicsOutput;
use iggy_cli::commands::binary_users::get_users::GetUsersOutput;
+use std::collections::BTreeSet;
#[derive(Debug, Clone, Copy, ValueEnum)]
pub(crate) enum ListMode {
@@ -152,3 +154,21 @@ pub(crate) fn parse_key_value(raw: &str) -> Result<(String, String), String> {
}
Ok((key.to_string(), value.to_string()))
}
+
+/// Split repeated `--reset` names into keys this build checks before sending
+/// and newer keys the server decides on.
+pub(crate) fn split_reset(names: &[String]) -> (BTreeSet<OptionKey>, BTreeSet<String>) {
+ let mut known = BTreeSet::new();
+ let mut newer = BTreeSet::new();
+ for name in names {
+ match name.parse::<OptionKey>() {
+ Ok(key) => {
+ known.insert(key);
+ }
+ Err(_) => {
+ newer.insert(name.clone());
+ }
+ }
+ }
+ (known, newer)
+}
diff --git a/core/cli/src/args/mod.rs b/core/cli/src/args/mod.rs
index a758284e2..ffe1145d4 100644
--- a/core/cli/src/args/mod.rs
+++ b/core/cli/src/args/mod.rs
@@ -173,11 +173,13 @@ pub(crate) enum Command {
/// client address as seen by the server and protocol type.
#[clap(verbatim_doc_comment)]
Me,
- /// list the options a resource's create command accepts
+ /// list the option catalog of a resource scope
///
/// Prints the server's option catalog for one scope (topic, stream, user):
- /// every key, the kind its default is reported in, that default, and the
- /// bounds each value is checked against. These are the keys `--set` takes.
+ /// every key, the kind its default is reported in, that default, and what a
+ /// client may do with it (create only, updatable, resettable, read only,
+ /// secret, critical). Create-only keys go to `--set` on topic create,
+ /// updatable keys to `--set` and `--reset` on topic and stream update.
#[clap(verbatim_doc_comment)]
Options(OptionsArgs),
/// get iggy server statistics
diff --git a/core/cli/src/args/stream.rs b/core/cli/src/args/stream.rs
index 1609b93f9..993c7294d 100644
--- a/core/cli/src/args/stream.rs
+++ b/core/cli/src/args/stream.rs
@@ -15,9 +15,10 @@
// specific language governing permissions and limitations
// under the License.
-use crate::args::common::ListMode;
+use crate::args::common::{ListMode, parse_key_value, split_reset};
use clap::{Args, Subcommand};
-use iggy::prelude::Identifier;
+use iggy::prelude::{Identifier, StreamUpdateOptions};
+use std::path::PathBuf;
#[derive(Debug, Clone, Subcommand)]
pub(crate) enum StreamAction {
@@ -28,6 +29,7 @@ pub(crate) enum StreamAction {
/// Examples:
/// iggy stream create prod
/// iggy stream create -s 1 test
+ /// iggy stream create --stream-key-file stream.key payments
#[clap(verbatim_doc_comment, visible_alias = "c")]
Create(StreamCreateArgs),
/// Delete stream with given ID
@@ -39,13 +41,14 @@ pub(crate) enum StreamAction {
/// iggy stream delete test
#[clap(verbatim_doc_comment, visible_alias = "d")]
Delete(StreamDeleteArgs),
- /// Update stream name for given stream ID
+ /// Update stream name or options for given stream ID
///
/// Stream ID can be specified as a stream name or ID
+ /// Only the settings passed change, every other setting keeps its value.
///
/// Examples:
- /// iggy stream update 1 production
- /// iggy stream update test development
+ /// iggy stream update 1 --name production
+ /// iggy stream update test --name development
#[clap(verbatim_doc_comment, visible_alias = "u")]
Update(StreamUpdateArgs),
/// Get details of a single stream with given ID
@@ -72,6 +75,15 @@ pub(crate) struct StreamCreateArgs {
/// Legacy stream ID flag (ignored)
#[clap(short, long)]
pub(crate) stream_id: Option<u32>,
+ /// File holding the stream's own at-rest encryption key
+ ///
+ /// Standard base64 of exactly 32 random bytes from a secure source, such
+ /// as "head -c 32 /dev/urandom | base64". Use a new key for every stream.
+ /// Without this flag the server encryption setting applies. The server
+ /// stores the key wrapped and reads report only its fingerprint. Unrelated
+ /// to --encryption-key, which encrypts message payloads on the client.
+ #[clap(long, value_name = "PATH", verbatim_doc_comment)]
+ pub(crate) stream_key_file: Option<PathBuf>,
/// Name of the stream
pub(crate) name: String,
}
@@ -91,7 +103,31 @@ pub(crate) struct StreamUpdateArgs {
/// Stream ID can be specified as a stream name or ID
pub(crate) stream_id: Identifier,
/// New name for the stream
- pub(crate) name: String,
+ #[arg(short, long)]
+ pub(crate) name: Option<String>,
+ /// Updatable stream option as key=value, repeatable
+ ///
+ /// Values are sent as strings and parsed server-side.
+ /// Run "iggy options stream" to list the keys and which ones can change.
+ #[arg(long = "set", value_name = "KEY=VALUE", value_parser = parse_key_value, verbatim_doc_comment)]
+ pub(crate) set: Vec<(String, String)>,
+ /// Setting to return to its default, repeatable
+ ///
+ /// The server refuses a key that cannot be reset.
+ #[arg(short, long, verbatim_doc_comment)]
+ pub(crate) reset: Vec<String>,
+}
+
+impl From<&StreamUpdateArgs> for StreamUpdateOptions {
+ fn from(args: &StreamUpdateArgs) -> Self {
+ let (reset, raw_reset) = split_reset(&args.reset);
+ StreamUpdateOptions {
+ name: args.name.clone(),
+ reset,
+ raw_reset,
+ raw: args.set.iter().cloned().collect(),
+ }
+ }
}
#[derive(Debug, Clone, Args)]
@@ -108,3 +144,39 @@ pub(crate) struct StreamListArgs {
#[clap(short, long, value_enum, default_value_t = ListMode::Table)]
pub(crate) list_mode: ListMode,
}
+
+#[cfg(test)]
+mod tests {
+ use super::StreamUpdateArgs;
+ use clap::Parser;
+ use iggy::prelude::StreamUpdateOptions;
+ use std::collections::{BTreeMap, BTreeSet};
+
+ #[derive(Parser)]
+ struct Update {
+ #[command(flatten)]
+ args: StreamUpdateArgs,
+ }
+
+ #[test]
+ fn given_stream_update_without_name_when_parsing_should_accept_reset_and_set() {
+ let parsed = Update::try_parse_from([
+ "iggy",
+ "orders",
+ "--reset",
+ "future_key",
+ "--set",
+ "newer_key=1",
+ ])
+ .expect("an update without a name parses");
+
+ assert_eq!(
+ StreamUpdateOptions::from(&parsed.args),
+ StreamUpdateOptions {
+ raw_reset: BTreeSet::from(["future_key".to_owned()]),
+ raw: BTreeMap::from([("newer_key".to_owned(), "1".to_owned())]),
+ ..StreamUpdateOptions::default()
+ }
+ );
+ }
+}
diff --git a/core/cli/src/args/topic.rs b/core/cli/src/args/topic.rs
index b79884804..8e400fe8d 100644
--- a/core/cli/src/args/topic.rs
+++ b/core/cli/src/args/topic.rs
@@ -15,9 +15,11 @@
// specific language governing permissions and limitations
// under the License.
-use crate::args::common::{ListMode, parse_key_value};
+use crate::args::common::{ListMode, parse_key_value, split_reset};
use clap::{Args, Subcommand};
-use iggy::prelude::{CompressionAlgorithm, Identifier, IggyExpiry, MaxTopicSize};
+use iggy::prelude::{
+ CompressionAlgorithm, Identifier, IggyExpiry, MaxTopicSize, TopicUpdateOptions,
+};
#[derive(Debug, Clone, Subcommand)]
pub(crate) enum TopicAction {
@@ -45,17 +47,19 @@ pub(crate) enum TopicAction {
/// iggy topic delete 2 debugs
#[clap(verbatim_doc_comment, visible_alias = "d")]
Delete(TopicDeleteArgs),
- /// Update topic name, compression algorithm and message expiry time for given topic ID in given stream ID
+ /// Update topic name, message expiry or max topic size for given topic ID in given stream ID
///
/// Stream ID can be specified as a stream name or ID
/// Topic ID can be specified as a topic name or ID
+ /// Only the settings passed change, every other setting keeps its value.
+ /// Compression, segment size, durability and the other create-time settings cannot change.
///
/// Examples
- /// iggy topic update 1 1 sensor3 none
- /// iggy topic update prod sensor3 old-sensor none
- /// iggy topic update test debugs ready gzip 15days
- /// iggy topic update 1 1 new-name gzip
- /// iggy topic update 1 2 new-name none 1day 1hour 1min 1sec
+ /// iggy topic update 1 1 --name sensor3
+ /// iggy topic update prod sensor3 --name old-sensor --message-expiry 15days
+ /// iggy topic update test debugs --max-topic-size 15GB
+ /// iggy topic update 1 2 --message-expiry 1day 1hour 1min 1sec
+ /// iggy topic update 1 2 --reset message_expiry --reset max_topic_size
#[clap(verbatim_doc_comment, visible_alias = "u")]
Update(TopicUpdateArgs),
/// Get topic detail for given topic ID and stream ID
@@ -152,21 +156,42 @@ pub(crate) struct TopicUpdateArgs {
#[arg(value_parser = clap::value_parser!(Identifier))]
pub(crate) topic_id: Identifier,
/// New name for the topic
- pub(crate) name: String,
- /// Compression metadata (none or gzip). Payload compression is not implemented
- #[arg(value_parser = clap::value_parser!(CompressionAlgorithm), verbatim_doc_comment)]
- pub(crate) compression_algorithm: CompressionAlgorithm,
+ #[arg(short, long)]
+ pub(crate) name: Option<String>,
/// New max topic size in human-readable format like "unlimited" or "15GB"
///
- /// Skipping this parameter or using "server_default" preserves the current max topic size.
/// A finite size cannot be lower than the topic segment size.
- #[arg(short, long, default_value = "server_default", verbatim_doc_comment)]
- pub(crate) max_topic_size: MaxTopicSize,
+ #[arg(short, long, verbatim_doc_comment)]
+ pub(crate) max_topic_size: Option<MaxTopicSize>,
/// New message expiry time in human-readable format like "unlimited" or "15days 2min 2s"
+ #[arg(short = 'e', long, num_args = 1.., value_parser = clap::value_parser!(IggyExpiry), verbatim_doc_comment)]
+ pub(crate) message_expiry: Option<Vec<IggyExpiry>>,
+ /// Updatable topic option as key=value, repeatable
///
- /// Skipping this parameter or using "server_default" preserves the current message expiry.
- #[arg(default_value = "server_default", value_parser = clap::value_parser!(IggyExpiry), verbatim_doc_comment)]
- pub(crate) message_expiry: Vec<IggyExpiry>,
+ /// Values are sent as strings and parsed server-side.
+ /// Run "iggy options topic" to list the keys and which ones can change.
+ #[arg(long = "set", value_name = "KEY=VALUE", value_parser = parse_key_value, verbatim_doc_comment)]
+ pub(crate) set: Vec<(String, String)>,
+ /// Setting to return to its default, like message_expiry or max_topic_size
+ ///
+ /// Repeat the flag to reset several settings.
+ /// The server refuses a key that cannot be reset.
+ #[arg(short, long, verbatim_doc_comment)]
+ pub(crate) reset: Vec<String>,
+}
+
+impl From<&TopicUpdateArgs> for TopicUpdateOptions {
+ fn from(args: &TopicUpdateArgs) -> Self {
+ let (reset, raw_reset) = split_reset(&args.reset);
+ TopicUpdateOptions {
+ name: args.name.clone(),
+ message_expiry: args.message_expiry.clone().map(IggyExpiry::from),
+ max_topic_size: args.max_topic_size,
+ reset,
+ raw_reset,
+ raw: args.set.iter().cloned().collect(),
+ }
+ }
}
#[derive(Debug, Clone, Args)]
@@ -196,6 +221,46 @@ pub(crate) struct TopicListArgs {
pub(crate) list_mode: ListMode,
}
+#[cfg(test)]
+mod update_tests {
+ use super::TopicUpdateArgs;
+ use clap::Parser;
+ use iggy::prelude::{OptionKey, TopicUpdateOptions};
+ use std::collections::{BTreeMap, BTreeSet};
+
+ #[derive(Parser)]
+ struct Update {
+ #[command(flatten)]
+ args: TopicUpdateArgs,
+ }
+
+ #[test]
+ fn given_reset_name_newer_than_build_when_parsing_topic_update_should_send_it_raw() {
+ let parsed = Update::try_parse_from([
+ "iggy",
+ "1",
+ "2",
+ "--reset",
+ "message_expiry",
+ "--reset",
+ "future_key",
+ "--set",
+ "newer_key=1",
+ ])
+ .expect("a reset of a newer key parses");
+
+ assert_eq!(
+ TopicUpdateOptions::from(&parsed.args),
+ TopicUpdateOptions {
+ reset: BTreeSet::from([OptionKey::MessageExpiry]),
+ raw_reset: BTreeSet::from(["future_key".to_owned()]),
+ raw: BTreeMap::from([("newer_key".to_owned(), "1".to_owned())]),
+ ..TopicUpdateOptions::default()
+ }
+ );
+ }
+}
+
#[cfg(test)]
mod durability_tests {
use super::TopicCreateArgs;
diff --git a/core/cli/src/commands/binary_streams/create_stream.rs b/core/cli/src/commands/binary_streams/create_stream.rs
index 5ab970799..53e7ebe43 100644
--- a/core/cli/src/commands/binary_streams/create_stream.rs
+++ b/core/cli/src/commands/binary_streams/create_stream.rs
@@ -18,24 +18,52 @@
use crate::commands::cli_command::{CliCommand, PRINT_TARGET};
use anyhow::Context;
use async_trait::async_trait;
-use iggy_common::Client;
-use iggy_common::create_stream::CreateStream;
+use iggy_common::{
+ Client, StreamCreateOptions, StreamDetails, StreamEncryptionKey, StreamEncryptionSource,
+};
+use std::fs;
+use std::path::PathBuf;
+use std::str::FromStr;
use tracing::{Level, event};
pub struct CreateStreamCmd {
- create_stream: CreateStream,
+ name: String,
+ stream_key_file: Option<PathBuf>,
}
impl CreateStreamCmd {
- pub fn new(name: String) -> Self {
+ pub fn new(name: String, stream_key_file: Option<PathBuf>) -> Self {
Self {
- create_stream: CreateStream { name },
+ name,
+ stream_key_file,
}
}
fn get_stream_id_info(&self) -> String {
"ID auto incremented".to_string()
}
+
+ /// Only a file named on this call supplies the key, so no key spreads across streams.
+ fn options(&self) -> anyhow::Result<StreamCreateOptions> {
+ let encryption_key = self
+ .stream_key_file
+ .as_ref()
+ .map(|path| {
+ let text = fs::read_to_string(path).with_context(|| {
+ format!(
+ "Problem reading stream encryption key file: {}",
+ path.display()
+ )
+ })?;
+ StreamEncryptionKey::from_str(text.trim_end())
+ .context("Stream encryption key must be standard base64 of exactly 32 bytes")
+ })
+ .transpose()?;
+ Ok(StreamCreateOptions {
... 93902 lines suppressed ...