Could Apache Host the Yahoo Cloud Serving Benchmark (YCSB)

[Jörg Domaschka <jo...@uni-ulm.de>]

Dear Apache Incubator PMC members,

the Yahoo Cloud Serving Benchmark (YCSB) is a benchmarking framework 
originally developed for comparing NoSQL DBMS, but by now used for all 
kinds of DBMS. It is well-established in industry and academia. 
Unfortunately, the original creator of the YCSB is no longer in charge 
of the project and the current maintainer only applies patches and bug 
fixes sporadically [1].

As a consequence, there is a lot of work going outside the main repo of 
the YCSB. Last time I checked there were more than 2,000 forks on 
github; not all of them are actively maintained and many of them do not 
even have a single commit.

Nevertheless, there are many different maintained versions of YCSB out 
there that often contain technically and methodologically incompatible 
changes. A consequence of this is that results reported from different 
experimenters based on YCSB are not compatible and cannot be compared 
with each other. Obviously, non-comparability is a worst case scenario 
for any benchmark and I think neither researchers nor practitioners can 
be happy about that situation (marketing people can, but this is a 
different story).

I think there needs to be a trustworthy source that maintains THE major 
YCSB version, takes care of integrating new DBMS versions and regularly 
releases new versions. I also think that the Apache Foundation is such a 
trustworthy source with established processes to negotiate between 
different stakeholders with different interests. Even more as YCSB is 
used for benchmarking Apache projects such as Cassandra, Ignite, and Hbase.

Thanks for reading, I am looking forward to your responses, feedback and 
thoughts.

Best regards,
  Jörg

[1] https://github.com/brianfrankcooper/YCSB

-- 
Dr. Jörg Domaschka
Lead Loosely Coupled Systems Group
Institute for Information Resource Management
Ulm University, Germany

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: Could Apache Host the Yahoo Cloud Serving Benchmark (YCSB)

[Nathan Hartman <ha...@gmail.com>]

On Thu, Feb 24, 2022 at 4:04 AM Jörg Domaschka <jo...@uni-ulm.de>
wrote:

> Dear Apache Incubator PMC members,
>
> the Yahoo Cloud Serving Benchmark (YCSB) is a benchmarking framework
> originally developed for comparing NoSQL DBMS, but by now used for all
> kinds of DBMS. It is well-established in industry and academia.
> Unfortunately, the original creator of the YCSB is no longer in charge
> of the project and the current maintainer only applies patches and bug
> fixes sporadically [1].
>
> As a consequence, there is a lot of work going outside the main repo of
> the YCSB. Last time I checked there were more than 2,000 forks on
> github; not all of them are actively maintained and many of them do not
> even have a single commit.
>
> Nevertheless, there are many different maintained versions of YCSB out
> there that often contain technically and methodologically incompatible
> changes. A consequence of this is that results reported from different
> experimenters based on YCSB are not compatible and cannot be compared
> with each other. Obviously, non-comparability is a worst case scenario
> for any benchmark and I think neither researchers nor practitioners can
> be happy about that situation (marketing people can, but this is a
> different story).
>
> I think there needs to be a trustworthy source that maintains THE major
> YCSB version, takes care of integrating new DBMS versions and regularly
> releases new versions. I also think that the Apache Foundation is such a
> trustworthy source with established processes to negotiate between
> different stakeholders with different interests. Even more as YCSB is
> used for benchmarking Apache projects such as Cassandra, Ignite, and Hbase.
>
> Thanks for reading, I am looking forward to your responses, feedback and
> thoughts.
>
> Best regards,
>   Jörg
>
> [1] https://github.com/brianfrankcooper/YCSB
>
> --
> Dr. Jörg Domaschka
> Lead Loosely Coupled Systems Group
> Institute for Information Resource Management
> Ulm University, Germany



Hello,

For a podling to be successful, it must have a community. Have you tried to
reach out to some of the other offshoots of YCSB to discuss the possibility
of building a consolidated community around YCSB in the form of a ASF
podling?

Cheers,
Nathan