You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by siddharth teotia <si...@gmail.com> on 2020/02/24 21:10:43 UTC
Jepsen testing for Pinot
I think at some point we should think about using Jepsen for safety-testing
of Pinot. While the tool has been predominantly used in databases that
have distributed transactions, extensive node communication, algorithms
relying on clock-skew, it can still be used to introduced faults in the
cluster and check for existence of problems.
I believe Helix and ZK are the perfect candidates (but that work is going
to be orthogonal to Pinot). However, since we use these extensively, it
might still be worthwhile to see if Jepsen framework can expose some
problems in Pinot.
https://jepsen.io/
Thanks,
Sidd
Re: Jepsen testing for Pinot
Posted by Subbu Subramaniam <mc...@apache.org>.
Certainly a possibility,
I am curious how they do it, though, Last time I tried to test corner cases, the only way I could reproduce them was by putting sleeps in the code, where some operations may take an unnaturally long time due to local implementation/thread contention/os issues, or such.
Being an external entity, you can test some things, but not all.
If they are allowing us to specifiy the protocol somehow, and run tests on it to prove correctness, that is one thing (reminds me of my grad school projects on protocol correctness). But that will still not guarantee pinot's implementation is correct ...
-Subbu
On 2020/02/25 15:09:49, Mayank Shrivastava <ma...@gmail.com> wrote:
> Agree on the derived systems part. One thing to explore though would be if this can be applied to the real-time segment commit protocol.
>
> Cheers,
> Mayank
>
> Sent from my iPhone
>
> > On Feb 24, 2020, at 1:17 PM, kishore g <g....@gmail.com> wrote:
> >
> > We should define a new set of tests for derived systems
> >
> >> On Mon, Feb 24, 2020 at 1:13 PM kishore g <g....@gmail.com> wrote:
> >>
> >> Does not make sense for Pinot
> >>
> >> On Mon, Feb 24, 2020 at 1:10 PM siddharth teotia <
> >> siddharthteotia@gmail.com> wrote:
> >>
> >>> I think at some point we should think about using Jepsen for
> >>> safety-testing
> >>> of Pinot. While the tool has been predominantly used in databases that
> >>> have distributed transactions, extensive node communication, algorithms
> >>> relying on clock-skew, it can still be used to introduced faults in the
> >>> cluster and check for existence of problems.
> >>>
> >>> I believe Helix and ZK are the perfect candidates (but that work is going
> >>> to be orthogonal to Pinot). However, since we use these extensively, it
> >>> might still be worthwhile to see if Jepsen framework can expose some
> >>> problems in Pinot.
> >>>
> >>> https://jepsen.io/
> >>>
> >>> Thanks,
> >>> Sidd
> >>>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org
> For additional commands, e-mail: dev-help@pinot.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org
For additional commands, e-mail: dev-help@pinot.apache.org
Re: Jepsen testing for Pinot
Posted by Mayank Shrivastava <ma...@gmail.com>.
Agree on the derived systems part. One thing to explore though would be if this can be applied to the real-time segment commit protocol.
Cheers,
Mayank
Sent from my iPhone
> On Feb 24, 2020, at 1:17 PM, kishore g <g....@gmail.com> wrote:
>
> We should define a new set of tests for derived systems
>
>> On Mon, Feb 24, 2020 at 1:13 PM kishore g <g....@gmail.com> wrote:
>>
>> Does not make sense for Pinot
>>
>> On Mon, Feb 24, 2020 at 1:10 PM siddharth teotia <
>> siddharthteotia@gmail.com> wrote:
>>
>>> I think at some point we should think about using Jepsen for
>>> safety-testing
>>> of Pinot. While the tool has been predominantly used in databases that
>>> have distributed transactions, extensive node communication, algorithms
>>> relying on clock-skew, it can still be used to introduced faults in the
>>> cluster and check for existence of problems.
>>>
>>> I believe Helix and ZK are the perfect candidates (but that work is going
>>> to be orthogonal to Pinot). However, since we use these extensively, it
>>> might still be worthwhile to see if Jepsen framework can expose some
>>> problems in Pinot.
>>>
>>> https://jepsen.io/
>>>
>>> Thanks,
>>> Sidd
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org
For additional commands, e-mail: dev-help@pinot.apache.org
Re: Jepsen testing for Pinot
Posted by kishore g <g....@gmail.com>.
We should define a new set of tests for derived systems
On Mon, Feb 24, 2020 at 1:13 PM kishore g <g....@gmail.com> wrote:
> Does not make sense for Pinot
>
> On Mon, Feb 24, 2020 at 1:10 PM siddharth teotia <
> siddharthteotia@gmail.com> wrote:
>
>> I think at some point we should think about using Jepsen for
>> safety-testing
>> of Pinot. While the tool has been predominantly used in databases that
>> have distributed transactions, extensive node communication, algorithms
>> relying on clock-skew, it can still be used to introduced faults in the
>> cluster and check for existence of problems.
>>
>> I believe Helix and ZK are the perfect candidates (but that work is going
>> to be orthogonal to Pinot). However, since we use these extensively, it
>> might still be worthwhile to see if Jepsen framework can expose some
>> problems in Pinot.
>>
>> https://jepsen.io/
>>
>> Thanks,
>> Sidd
>>
>
Re: Jepsen testing for Pinot
Posted by kishore g <g....@gmail.com>.
Does not make sense for Pinot
On Mon, Feb 24, 2020 at 1:10 PM siddharth teotia <si...@gmail.com>
wrote:
> I think at some point we should think about using Jepsen for safety-testing
> of Pinot. While the tool has been predominantly used in databases that
> have distributed transactions, extensive node communication, algorithms
> relying on clock-skew, it can still be used to introduced faults in the
> cluster and check for existence of problems.
>
> I believe Helix and ZK are the perfect candidates (but that work is going
> to be orthogonal to Pinot). However, since we use these extensively, it
> might still be worthwhile to see if Jepsen framework can expose some
> problems in Pinot.
>
> https://jepsen.io/
>
> Thanks,
> Sidd
>