You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "adoroszlai (via GitHub)" <gi...@apache.org> on 2026/03/06 09:28:42 UTC
[PR] HDDS-14771. Split server-side load testers from freon [ozone]
adoroszlai opened a new pull request, #9874:
URL: https://github.com/apache/ozone/pull/9874
## What changes were proposed in this pull request?
Freon is a multi-purpose utility for load-testing Ozone. Most subcommands exercise public API, but some use server-side code (e.g. to simulate some component). Thus `ozone-freon` has dependency on multiple Ozone server components.
In HDDS-14439 we are moving to Java 17 for server components, but keeping Java 8 for client components. The goal of this change is to make Freon a client component, thereby keep it usable with Java 8 in the future. It eliminates Freon's dependency on server components by splitting some of its generators into a new module `ozone-vapor`, making Freon a pure client-side tool.
We cannot use the top-level command `ozone freon` for both sets of commands, because subcommands share the same classpath. Therefore we also have to introduce a new top-level command (`ozone vapor`). These subcommands are rather obscure, so I think only mentioning this change in the release notes is OK.
`BaseFreonGenerator` is shared, Subcommands of `ozone vapor` need to
Package name is kept to reduce change.
https://issues.apache.org/jira/browse/HDDS-14771
## How was this patch tested?
```bash
$ ozone vapor --help
Usage: ozone vapor [-hV] [--server] [--verbose] [-conf=<configurationPath>]
[-D=<String=String>]... [COMMAND]
Load generator using Ozone server components
...
Commands:
cgdn Offline container metadata generator for
Ozone Datanodes.
cgom Offline container metadata generator for
Ozone Manager
cgscm Offline container metadata generator for
Storage Container Manager
cmdw, chunk-manager-disk-write Write chunks as fast as possible.
cr, container-replicator Replicate / download closed containers.
falg, follower-append-log-generator Generate append log entries to a
follower server
lalg, leader-append-log-generator Generate append log entries to a leader
server
scm-throughput-benchmark, stb Benchmark for scm throughput.
simulate-datanode Simulate one or many datanodes and
register them to SCM.This is used to
stress test SCM handling a massive
cluster.
strmg, streaming-generator Create directory structure and stream
them multiple times.
```
Existing tests cover Freon.
https://github.com/adoroszlai/ozone/actions/runs/22756767625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14771. Split server-side load testers from freon [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #9874:
URL: https://github.com/apache/ozone/pull/9874#discussion_r2901989117
##########
hadoop-ozone/vapor/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/GeneratorDatanode.java:
##########
@@ -74,13 +74,13 @@
optionListHeading =
"\nExecute this command with different parameters for each datanodes. "
+ "For example if you have 10 datanodes, use "
- + "'ozone freon cgdn --index=1 --datanodes=10', 'ozone freon"
- + " cgdn --index=2 --datanodes=10', 'ozone freon cgdn "
+ + "'ozone vapor cgdn --index=1 --datanodes=10', 'ozone vapor"
+ + " cgdn --index=2 --datanodes=10', 'ozone vapor cgdn "
Review Comment:
@adoroszlai , thanks for the update!
Let's print a warning when a vapor subcommand is run by `ozone freon`. Something likes
```
WARN: The command `ozone freon ockv` is changed to `ozone vapor ockv`.
Please update your scripts since `ozone freon ockv` may no longer
work in future releases.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14771. Split server-side load testers from freon [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #9874:
URL: https://github.com/apache/ozone/pull/9874#discussion_r2901860336
##########
hadoop-ozone/vapor/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/GeneratorDatanode.java:
##########
@@ -74,13 +74,13 @@
optionListHeading =
"\nExecute this command with different parameters for each datanodes. "
+ "For example if you have 10 datanodes, use "
- + "'ozone freon cgdn --index=1 --datanodes=10', 'ozone freon"
- + " cgdn --index=2 --datanodes=10', 'ozone freon cgdn "
+ + "'ozone vapor cgdn --index=1 --datanodes=10', 'ozone vapor"
+ + " cgdn --index=2 --datanodes=10', 'ozone vapor cgdn "
Review Comment:
Added handling for compatibility in `ozone` shell script. Thanks for the idea.
```bash
$ ozone freon cr --help
Usage: ozone vapor cr [-fhV] [--verbose] [--datanode=<datanode>]
[--duration=<duration>] [-n=<testNo>] [-p=<prefix>]
[-t=<threadNo>]
Replicate / download closed containers.
--datanode=<datanode>
Replicate only containers on this specific datanode.
Default:
...
```
```bash
$ ozone freon ockv --help
Usage: ozone freon ockv [-fhsV] [--verbose] [-b=<bucketName>]
[--duration=<duration>] [-n=<testNo>]
[--om-service-id=<omServiceID>] [-p=<prefix>]
[-t=<threadNo>] [-v=<volumeName>]
Validate keys with the help of the ozone clients.
-b, --bucket=<bucketName>
Name of the bucket which contains the test data.
Default: bucket1
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14771. Split server-side load testers from freon [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #9874:
URL: https://github.com/apache/ozone/pull/9874#discussion_r2901120528
##########
hadoop-ozone/vapor/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/GeneratorDatanode.java:
##########
@@ -74,13 +74,13 @@
optionListHeading =
"\nExecute this command with different parameters for each datanodes. "
+ "For example if you have 10 datanodes, use "
- + "'ozone freon cgdn --index=1 --datanodes=10', 'ozone freon"
- + " cgdn --index=2 --datanodes=10', 'ozone freon cgdn "
+ + "'ozone vapor cgdn --index=1 --datanodes=10', 'ozone vapor"
+ + " cgdn --index=2 --datanodes=10', 'ozone vapor cgdn "
Review Comment:
@adoroszlai , Thanks for working on this! It is good to separate the client and server commands.
Some of the "ozone freon" subcommands become "ozone vapor". It is a kind of an incompatible change. Is it possible to make those subcommands work for both "ozone freon" and "ozone vapor"? E.g. both of the following will work:
- ozone freon cgdn --index=1 --datanodes=10
- ozone vapor cgdn --index=1 --datanodes=10
We may remove "ozone freon" later on for those subcommands.
##########
hadoop-ozone/vapor/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/GeneratorDatanode.java:
##########
@@ -74,13 +74,13 @@
optionListHeading =
"\nExecute this command with different parameters for each datanodes. "
+ "For example if you have 10 datanodes, use "
- + "'ozone freon cgdn --index=1 --datanodes=10', 'ozone freon"
- + " cgdn --index=2 --datanodes=10', 'ozone freon cgdn "
+ + "'ozone vapor cgdn --index=1 --datanodes=10', 'ozone vapor"
+ + " cgdn --index=2 --datanodes=10', 'ozone vapor cgdn "
Review Comment:
@adoroszlai , Thanks for working on this! It is good to separate the client and server commands.
Some of the "ozone freon" subcommands become "ozone vapor". It is a kind of an incompatible change. Is it possible to make those subcommands work for both "ozone freon" and "ozone vapor"? E.g. both of the following will work:
- ozone freon cgdn --index=1 --datanodes=10
- ozone vapor cgdn --index=1 --datanodes=10
We will remove "ozone freon" later on for those subcommands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14771. Split server-side load testers from freon [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #9874:
URL: https://github.com/apache/ozone/pull/9874#issuecomment-4012779942
> client components should be built using separate target and be published as a separate set of artifacts that can be used by different products
They are separate artifacts, which other products can depend on.
Client components can also be used from "edge" nodes, where only client-side artifacts are present.
Parts of Ozone CLI (`ozone sh`, `ozone fs`, `ozone admin`, etc.) are also considered client components.
> what is the intended Freon usage outside of Ozone deployment?
Freon is used in tests for at least 3 reasons:
- exercise code paths currently only available via Java API, not via regular CLI (this is the hsync scenario I ran into)
- as a shortcut for multiple statements (create volume, create bucket, create key in one step)
- exercise Freon itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14771. Split server-side load testers from freon [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #9874:
URL: https://github.com/apache/ozone/pull/9874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14771. Split server-side load testers from freon [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #9874:
URL: https://github.com/apache/ozone/pull/9874#discussion_r2901120528
##########
hadoop-ozone/vapor/src/main/java/org/apache/hadoop/ozone/freon/containergenerator/GeneratorDatanode.java:
##########
@@ -74,13 +74,13 @@
optionListHeading =
"\nExecute this command with different parameters for each datanodes. "
+ "For example if you have 10 datanodes, use "
- + "'ozone freon cgdn --index=1 --datanodes=10', 'ozone freon"
- + " cgdn --index=2 --datanodes=10', 'ozone freon cgdn "
+ + "'ozone vapor cgdn --index=1 --datanodes=10', 'ozone vapor"
+ + " cgdn --index=2 --datanodes=10', 'ozone vapor cgdn "
Review Comment:
@adoroszlai , Thanks for working on this! It is good to separate the client and server commands.
Some of the "ozone freon" subcommands become "ozone vapor". It is a kind of an incompatible change. Is it possible to make those subcommands work for both "ozone freon" and "ozone vapor"? E.g. both of the following will work:
- ozone freon cgdn --index=1 --datanodes=10
- ozone vapor cgdn --index=1 --datanodes=10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14771. Split server-side load testers from freon [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #9874:
URL: https://github.com/apache/ozone/pull/9874#issuecomment-4021798935
Thanks @szetszwo, @yandrey321 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14771. Split server-side load testers from freon [ozone]
Posted by "yandrey321 (via GitHub)" <gi...@apache.org>.
yandrey321 commented on PR #9874:
URL: https://github.com/apache/ozone/pull/9874#issuecomment-4012187461
I think we should consider freon as a server component since its a part of Ozone deployment.
IMHO the client components should be built using separate target and be published as a separate set of artifacts that can be used by different products.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org