You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by "Petter von Dolwitz (Hem)" <pe...@gmail.com> on 2017/12/05 18:42:25 UTC
Data inconsistency after restart
Hi Kudu users,
We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline for
evaluation we ingested 3 month worth of data. During ingestion we were
facing messages from the maintenance threads that a soft memory limit were
reached. It seems like the background maintenance threads stopped
performing their tasks at this point in time. It also so seems like the
memory was never recovered even after stopping ingestion so I guess there
was a large backlog being built up. I guess the root cause here is that we
were a bit too conservative when giving Kudu memory. After a reststart a
lot of maintenance tasks were started (i.e. compaction).
When we verified that all data was inserted we found that some data was
missing. We added this missing data and on some chunks we got the
information that all rows were already present, i.e impala says something
like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
shows that the Kudu table is complete. So, even though we did not insert
any data on some chunks, a count(*) operation over these chunks now returns
a different value.
Now to my question. Will data be inconsistent if we recycle Kudu after
seeing soft memory limit warnings?
Is there a way to tell when it is safe to restart Kudu to avoid these
issues? Should we use any special procedure when restarting (e.g. only
restart the tablet servers, only restart one tablet server at a time or
something like that)?
The table design uses 50 tablets per day (times 90 days). It is 8 TB of
data after 3xreplication over 5 tablet servers.
Thanks,
Petter
Re: Data inconsistency after restart
Posted by "Petter von Dolwitz (Hem)" <pe...@gmail.com>.
Hi Todd,
I have not set aside time to re-create the issue again. I still aim to do
this. When I do, I will collect all relevant logs and create a JIRA.
Thank you for following up.
Br,
Petter
2018-01-05 0:35 GMT+01:00 Todd Lipcon <to...@cloudera.com>:
> Hey Petter,
>
> Did you ever get to the bottom of this? We definitely don't expect Kudu to
> lose data on a restart (and we have hundreds of tests running continuously
> which try to ensure this)
>
> -Todd
>
> On Fri, Dec 8, 2017 at 10:13 PM, David Alves <da...@gmail.com>
> wrote:
>
>> Hi Petter
>>
>> Don't have answers yet, but I do have some more questions (inline)
>>
>> Petter von Dolwitz (Hem) writes:
>>
>> Hi David,
>>>
>>> In short to summarize:
>>>
>>> 1. I ingest data. Kudus maintenance threads stops working (soft memory
>>> limit) and incoming data is throttled. There are no errors reported on
>>> the
>>> client side.
>>>
>> What is the "client side"? impala? spark? java/c++.
>>
>> 2. I stop ingestion and wait until i *think* Kudu is finsished.
>>>
>> The question above is pertinent. Impala will not return until a query
>> is fully successful, though it may return an error and leave a query
>> only half-way executed. If you're using the client apis directly
>> though are you checking for error when inserting?
>>
>> 3. I restart Kudu.
>>> 4. I validate the inserted data by doing count(*) on groups of data in
>>> Kudu. For several groups, Kudu reports a lot of rows missing.
>>>
>> Kudu's default scan mode is READ_LATEST. While this is the most
>> performance oriented mode, its also the one with the least guarantees
>> so, on startup its possible that it reads from a stale replica, giving
>> the _appearance_ that rows went missing. Things to try here:
>> - Try the same query a few minutes later. Is the answer different?
>> - If the above is true consider changing your scan mode to
>> READ_AT_SNAPHOT. In this mode data is guaranteed not to be state,
>> though you might have to wait for all replicas to be ready
>>
>> 5. I ingest the same data again. Client reports that all row are already
>>> present.
>>>
>> This isn't surprising _if_ the problem is indeed from state replicas.
>>
>>> 6. Doing the count(*) exercise again now gives me the correct number of
>>> rows.
>>>
>>> This tells me that the data was ingested into Kudu on the first attempt
>>> but
>>> a scan did not find the data. Inserting the data again made it
>>> visible.
>>>
>> Can it be that after the scan it's just that enough time has elapsed
>> so that all replicas are caught up? I'd say this is likely the case.
>>
>>>
>>> Br,
>>> Petter
>>>
>>> 2017-12-07 21:39 GMT+01:00 David Alves <da...@gmail.com>:
>>>
>>> Hi Petter
>>>>
>>>> I'd like to clarify what exactly happened and exactly what are you
>>>> referring to as "inconsistency".
>>>> From what I understand of the first error you observed, the Kudu
>>>> was
>>>> underprovisioned, memory wise, and the ingest jobs/queries failed. Is
>>>> that
>>>> right? Since Kudu doesn't have atomic multi-row writes, it's currently
>>>> expected in this case that you'll end up with partially written data.
>>>> If you tried the same job again, and it succeeded, for certain
>>>> types of
>>>> operation (UPSERT, INSERT IGNORE) then the remaining rows would be
>>>> written
>>>> and all the data would be there as expected.
>>>> I'd like to distinguish this lack of atomicity on multi-row
>>>> transactions from "inconsistency", which is what you might observe if an
>>>> operation didn't fail, but you couldn't see all the data. For this
>>>> latter
>>>> case there are options you can choose to avoid any inconsistency.
>>>>
>>>> Best
>>>> David
>>>>
>>>>
>>>>
>>>> On Wed, Dec 6, 2017 at 4:26 AM, Petter von Dolwitz (Hem) <
>>>> petter.von.dolwitz@gmail.com> wrote:
>>>>
>>>> Thanks for your reply Andrew!
>>>>>
>>>>> >How did you verify that all the data was inserted and how did >you
>>>>> find
>>>>> some data missing?
>>>>> This was done using Impala. We counted the rows for groups representing
>>>>> the chunks we inserted.
>>>>>
>>>>> >Following up on what I posted, take a look at
>>>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>>>> read_operations_scans. It seems definitely possible that not all of the
>>>>> rows had finished inserting when counting, or that the scans were sent
>>>>> to a
>>>>> stale replica.
>>>>> Before we shut down we could only see the following in the logs. I.e.,
>>>>> no
>>>>> sign that ingestion was still ongoing.
>>>>>
>>>>> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
>>>>> 07:27:35.010694 90793 maintenance_manager.cc:383] P
>>>>> a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft memory
>>>>> limit
>>>>> (current capacity is 67.52%). However, there are no ops currently
>>>>> runnable
>>>>> which would free memory.
>>>>>
>>>>> Also the (cloudera) metric total_kudu_rows_inserted_rate_
>>>>> across_kudu_replicas
>>>>> showed zero.
>>>>>
>>>>> Still it seems like some data became inconsistent after restart. But if
>>>>> the maintenance_manager performs important jobs that are required to
>>>>> ensure
>>>>> that all data is inserted then I can understand why we ended up with
>>>>> inconsistent data. But, if I understand you correct, you are saying
>>>>> that
>>>>> these jobs are not critical for ingestion. In the link you provided I
>>>>> read
>>>>> "Impala scans are currently performed as READ_LATEST and have no
>>>>> consistency guarantees.". I would assume this means that it does not
>>>>> guarantee consistency if new data is inserted but should give valid
>>>>> (and
>>>>> same) results if no new data is inserted?
>>>>>
>>>>> I have not tried the ksck tool yet. Thank you for reminding. I will
>>>>> have
>>>>> a look.
>>>>>
>>>>> Br,
>>>>> Petter
>>>>>
>>>>>
>>>>> 2017-12-06 1:31 GMT+01:00 Andrew Wong <aw...@cloudera.com>:
>>>>>
>>>>> How did you verify that all the data was inserted and how did you find
>>>>>>
>>>>>>> some data missing? I'm wondering if it's possible that the initial
>>>>>>> "missing" data was data that Kudu was still in the process of
>>>>>>> inserting
>>>>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>>>>
>>>>>>>
>>>>>> Following up on what I posted, take a look at
>>>>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>>>>> read_operations_scans. It seems definitely possible that not all of
>>>>>> the
>>>>>> rows had finished inserting when counting, or that the scans were
>>>>>> sent to a
>>>>>> stale replica.
>>>>>>
>>>>>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong <aw...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Petter,
>>>>>>>
>>>>>>> When we verified that all data was inserted we found that some data
>>>>>>> was
>>>>>>>
>>>>>>>> missing. We added this missing data and on some chunks we got the
>>>>>>>> information that all rows were already present, i.e impala says
>>>>>>>> something
>>>>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again
>>>>>>>> now
>>>>>>>> shows that the Kudu table is complete. So, even though we did not
>>>>>>>> insert
>>>>>>>> any data on some chunks, a count(*) operation over these chunks now
>>>>>>>> returns
>>>>>>>> a different value.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> How did you verify that all the data was inserted and how did you
>>>>>>> find
>>>>>>> some data missing? I'm wondering if it's possible that the initial
>>>>>>> "missing" data was data that Kudu was still in the process of
>>>>>>> inserting
>>>>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>>>>
>>>>>>> Now to my question. Will data be inconsistent if we recycle Kudu
>>>>>>> after
>>>>>>>
>>>>>>>> seeing soft memory limit warnings?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Your data should be consistently written, even with those warnings.
>>>>>>> AFAIK they would cause a bit of slowness, not incorrect results.
>>>>>>>
>>>>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>>>>>
>>>>>>>> issues? Should we use any special procedure when restarting (e.g.
>>>>>>>> only
>>>>>>>> restart the tablet servers, only restart one tablet server at a
>>>>>>>> time or
>>>>>>>> something like that)?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> In general, you can use the `ksck` tool to check the health of your
>>>>>>> cluster. See https://kudu.apache.org/docs/c
>>>>>>> ommand_line_tools_referenc
>>>>>>> e.html#cluster-ksck for more details. For restarting a cluster, I
>>>>>>> would recommend taking down all tablet servers at once, otherwise
>>>>>>> tablet
>>>>>>> replicas may try to replicate data from the server that was taken
>>>>>>> down.
>>>>>>>
>>>>>>> Hope this helped,
>>>>>>> Andrew
>>>>>>>
>>>>>>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
>>>>>>> petter.von.dolwitz@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Kudu users,
>>>>>>>>
>>>>>>>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline
>>>>>>>> for
>>>>>>>> evaluation we ingested 3 month worth of data. During ingestion we
>>>>>>>> were
>>>>>>>> facing messages from the maintenance threads that a soft memory
>>>>>>>> limit were
>>>>>>>> reached. It seems like the background maintenance threads stopped
>>>>>>>> performing their tasks at this point in time. It also so seems like
>>>>>>>> the
>>>>>>>> memory was never recovered even after stopping ingestion so I guess
>>>>>>>> there
>>>>>>>> was a large backlog being built up. I guess the root cause here is
>>>>>>>> that we
>>>>>>>> were a bit too conservative when giving Kudu memory. After a
>>>>>>>> reststart a
>>>>>>>> lot of maintenance tasks were started (i.e. compaction).
>>>>>>>>
>>>>>>>> When we verified that all data was inserted we found that some data
>>>>>>>> was missing. We added this missing data and on some chunks we got
>>>>>>>> the
>>>>>>>> information that all rows were already present, i.e impala says
>>>>>>>> something
>>>>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again
>>>>>>>> now
>>>>>>>> shows that the Kudu table is complete. So, even though we did not
>>>>>>>> insert
>>>>>>>> any data on some chunks, a count(*) operation over these chunks now
>>>>>>>> returns
>>>>>>>> a different value.
>>>>>>>>
>>>>>>>> Now to my question. Will data be inconsistent if we recycle Kudu
>>>>>>>> after
>>>>>>>> seeing soft memory limit warnings?
>>>>>>>>
>>>>>>>> Is there a way to tell when it is safe to restart Kudu to avoid
>>>>>>>> these
>>>>>>>> issues? Should we use any special procedure when restarting (e.g.
>>>>>>>> only
>>>>>>>> restart the tablet servers, only restart one tablet server at a
>>>>>>>> time or
>>>>>>>> something like that)?
>>>>>>>>
>>>>>>>> The table design uses 50 tablets per day (times 90 days). It is 8 TB
>>>>>>>> of data after 3xreplication over 5 tablet servers.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Petter
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Andrew Wong
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Andrew Wong
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>
>> --
>> David Alves
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Data inconsistency after restart
Posted by Todd Lipcon <to...@cloudera.com>.
Hey Petter,
Did you ever get to the bottom of this? We definitely don't expect Kudu to
lose data on a restart (and we have hundreds of tests running continuously
which try to ensure this)
-Todd
On Fri, Dec 8, 2017 at 10:13 PM, David Alves <da...@gmail.com> wrote:
> Hi Petter
>
> Don't have answers yet, but I do have some more questions (inline)
>
> Petter von Dolwitz (Hem) writes:
>
> Hi David,
>>
>> In short to summarize:
>>
>> 1. I ingest data. Kudus maintenance threads stops working (soft memory
>> limit) and incoming data is throttled. There are no errors reported on the
>> client side.
>>
> What is the "client side"? impala? spark? java/c++.
>
> 2. I stop ingestion and wait until i *think* Kudu is finsished.
>>
> The question above is pertinent. Impala will not return until a query
> is fully successful, though it may return an error and leave a query
> only half-way executed. If you're using the client apis directly
> though are you checking for error when inserting?
>
> 3. I restart Kudu.
>> 4. I validate the inserted data by doing count(*) on groups of data in
>> Kudu. For several groups, Kudu reports a lot of rows missing.
>>
> Kudu's default scan mode is READ_LATEST. While this is the most
> performance oriented mode, its also the one with the least guarantees
> so, on startup its possible that it reads from a stale replica, giving
> the _appearance_ that rows went missing. Things to try here:
> - Try the same query a few minutes later. Is the answer different?
> - If the above is true consider changing your scan mode to
> READ_AT_SNAPHOT. In this mode data is guaranteed not to be state,
> though you might have to wait for all replicas to be ready
>
> 5. I ingest the same data again. Client reports that all row are already
>> present.
>>
> This isn't surprising _if_ the problem is indeed from state replicas.
>
>> 6. Doing the count(*) exercise again now gives me the correct number of
>> rows.
>>
>> This tells me that the data was ingested into Kudu on the first attempt
>> but
>> a scan did not find the data. Inserting the data again made it
>> visible.
>>
> Can it be that after the scan it's just that enough time has elapsed
> so that all replicas are caught up? I'd say this is likely the case.
>
>>
>> Br,
>> Petter
>>
>> 2017-12-07 21:39 GMT+01:00 David Alves <da...@gmail.com>:
>>
>> Hi Petter
>>>
>>> I'd like to clarify what exactly happened and exactly what are you
>>> referring to as "inconsistency".
>>> From what I understand of the first error you observed, the Kudu
>>> was
>>> underprovisioned, memory wise, and the ingest jobs/queries failed. Is
>>> that
>>> right? Since Kudu doesn't have atomic multi-row writes, it's currently
>>> expected in this case that you'll end up with partially written data.
>>> If you tried the same job again, and it succeeded, for certain
>>> types of
>>> operation (UPSERT, INSERT IGNORE) then the remaining rows would be
>>> written
>>> and all the data would be there as expected.
>>> I'd like to distinguish this lack of atomicity on multi-row
>>> transactions from "inconsistency", which is what you might observe if an
>>> operation didn't fail, but you couldn't see all the data. For this latter
>>> case there are options you can choose to avoid any inconsistency.
>>>
>>> Best
>>> David
>>>
>>>
>>>
>>> On Wed, Dec 6, 2017 at 4:26 AM, Petter von Dolwitz (Hem) <
>>> petter.von.dolwitz@gmail.com> wrote:
>>>
>>> Thanks for your reply Andrew!
>>>>
>>>> >How did you verify that all the data was inserted and how did >you find
>>>> some data missing?
>>>> This was done using Impala. We counted the rows for groups representing
>>>> the chunks we inserted.
>>>>
>>>> >Following up on what I posted, take a look at
>>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>>> read_operations_scans. It seems definitely possible that not all of the
>>>> rows had finished inserting when counting, or that the scans were sent
>>>> to a
>>>> stale replica.
>>>> Before we shut down we could only see the following in the logs. I.e.,
>>>> no
>>>> sign that ingestion was still ongoing.
>>>>
>>>> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
>>>> 07:27:35.010694 90793 maintenance_manager.cc:383] P
>>>> a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft memory
>>>> limit
>>>> (current capacity is 67.52%). However, there are no ops currently
>>>> runnable
>>>> which would free memory.
>>>>
>>>> Also the (cloudera) metric total_kudu_rows_inserted_rate_
>>>> across_kudu_replicas
>>>> showed zero.
>>>>
>>>> Still it seems like some data became inconsistent after restart. But if
>>>> the maintenance_manager performs important jobs that are required to
>>>> ensure
>>>> that all data is inserted then I can understand why we ended up with
>>>> inconsistent data. But, if I understand you correct, you are saying
>>>> that
>>>> these jobs are not critical for ingestion. In the link you provided I
>>>> read
>>>> "Impala scans are currently performed as READ_LATEST and have no
>>>> consistency guarantees.". I would assume this means that it does not
>>>> guarantee consistency if new data is inserted but should give valid (and
>>>> same) results if no new data is inserted?
>>>>
>>>> I have not tried the ksck tool yet. Thank you for reminding. I will have
>>>> a look.
>>>>
>>>> Br,
>>>> Petter
>>>>
>>>>
>>>> 2017-12-06 1:31 GMT+01:00 Andrew Wong <aw...@cloudera.com>:
>>>>
>>>> How did you verify that all the data was inserted and how did you find
>>>>>
>>>>>> some data missing? I'm wondering if it's possible that the initial
>>>>>> "missing" data was data that Kudu was still in the process of
>>>>>> inserting
>>>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>>>
>>>>>>
>>>>> Following up on what I posted, take a look at
>>>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>>>> read_operations_scans. It seems definitely possible that not all of the
>>>>> rows had finished inserting when counting, or that the scans were sent
>>>>> to a
>>>>> stale replica.
>>>>>
>>>>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong <aw...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>> Hi Petter,
>>>>>>
>>>>>> When we verified that all data was inserted we found that some data
>>>>>> was
>>>>>>
>>>>>>> missing. We added this missing data and on some chunks we got the
>>>>>>> information that all rows were already present, i.e impala says
>>>>>>> something
>>>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again
>>>>>>> now
>>>>>>> shows that the Kudu table is complete. So, even though we did not
>>>>>>> insert
>>>>>>> any data on some chunks, a count(*) operation over these chunks now
>>>>>>> returns
>>>>>>> a different value.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> How did you verify that all the data was inserted and how did you find
>>>>>> some data missing? I'm wondering if it's possible that the initial
>>>>>> "missing" data was data that Kudu was still in the process of
>>>>>> inserting
>>>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>>>
>>>>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>>>>>
>>>>>>> seeing soft memory limit warnings?
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Your data should be consistently written, even with those warnings.
>>>>>> AFAIK they would cause a bit of slowness, not incorrect results.
>>>>>>
>>>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>>>>
>>>>>>> issues? Should we use any special procedure when restarting (e.g.
>>>>>>> only
>>>>>>> restart the tablet servers, only restart one tablet server at a time
>>>>>>> or
>>>>>>> something like that)?
>>>>>>>
>>>>>>
>>>>>>
>>>>>> In general, you can use the `ksck` tool to check the health of your
>>>>>> cluster. See https://kudu.apache.org/docs/command_line_tools_referenc
>>>>>> e.html#cluster-ksck for more details. For restarting a cluster, I
>>>>>> would recommend taking down all tablet servers at once, otherwise
>>>>>> tablet
>>>>>> replicas may try to replicate data from the server that was taken
>>>>>> down.
>>>>>>
>>>>>> Hope this helped,
>>>>>> Andrew
>>>>>>
>>>>>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
>>>>>> petter.von.dolwitz@gmail.com> wrote:
>>>>>>
>>>>>> Hi Kudu users,
>>>>>>>
>>>>>>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline for
>>>>>>> evaluation we ingested 3 month worth of data. During ingestion we
>>>>>>> were
>>>>>>> facing messages from the maintenance threads that a soft memory
>>>>>>> limit were
>>>>>>> reached. It seems like the background maintenance threads stopped
>>>>>>> performing their tasks at this point in time. It also so seems like
>>>>>>> the
>>>>>>> memory was never recovered even after stopping ingestion so I guess
>>>>>>> there
>>>>>>> was a large backlog being built up. I guess the root cause here is
>>>>>>> that we
>>>>>>> were a bit too conservative when giving Kudu memory. After a
>>>>>>> reststart a
>>>>>>> lot of maintenance tasks were started (i.e. compaction).
>>>>>>>
>>>>>>> When we verified that all data was inserted we found that some data
>>>>>>> was missing. We added this missing data and on some chunks we got the
>>>>>>> information that all rows were already present, i.e impala says
>>>>>>> something
>>>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again
>>>>>>> now
>>>>>>> shows that the Kudu table is complete. So, even though we did not
>>>>>>> insert
>>>>>>> any data on some chunks, a count(*) operation over these chunks now
>>>>>>> returns
>>>>>>> a different value.
>>>>>>>
>>>>>>> Now to my question. Will data be inconsistent if we recycle Kudu
>>>>>>> after
>>>>>>> seeing soft memory limit warnings?
>>>>>>>
>>>>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>>>>> issues? Should we use any special procedure when restarting (e.g.
>>>>>>> only
>>>>>>> restart the tablet servers, only restart one tablet server at a time
>>>>>>> or
>>>>>>> something like that)?
>>>>>>>
>>>>>>> The table design uses 50 tablets per day (times 90 days). It is 8 TB
>>>>>>> of data after 3xreplication over 5 tablet servers.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Petter
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Andrew Wong
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Andrew Wong
>>>>>
>>>>>
>>>>
>>>>
>>>
>
> --
> David Alves
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Data inconsistency after restart
Posted by David Alves <da...@gmail.com>.
Hi Petter
Don't have answers yet, but I do have some more questions
(inline)
Petter von Dolwitz (Hem) writes:
> Hi David,
>
> In short to summarize:
>
> 1. I ingest data. Kudus maintenance threads stops working (soft
> memory
> limit) and incoming data is throttled. There are no errors
> reported on the
> client side.
What is the "client side"? impala? spark? java/c++.
> 2. I stop ingestion and wait until i *think* Kudu is finsished.
The question above is pertinent. Impala will not return until a
query
is fully successful, though it may return an error and leave a
query
only half-way executed. If you're using the client apis directly
though are you checking for error when inserting?
> 3. I restart Kudu.
> 4. I validate the inserted data by doing count(*) on groups of
> data in
> Kudu. For several groups, Kudu reports a lot of rows missing.
Kudu's default scan mode is READ_LATEST. While this is the most
performance oriented mode, its also the one with the least
guarantees
so, on startup its possible that it reads from a stale replica,
giving
the _appearance_ that rows went missing. Things to try here:
- Try the same query a few minutes later. Is the answer
different?
- If the above is true consider changing your scan mode to
READ_AT_SNAPHOT. In this mode data is guaranteed not to be
state,
though you might have to wait for all replicas to be ready
> 5. I ingest the same data again. Client reports that all row are
> already
> present.
This isn't surprising _if_ the problem is indeed from state
replicas.
> 6. Doing the count(*) exercise again now gives me the correct
> number of
> rows.
>
> This tells me that the data was ingested into Kudu on the first
> attempt but
> a scan did not find the data. Inserting the data again made it
> visible.
Can it be that after the scan it's just that enough time has
elapsed
so that all replicas are caught up? I'd say this is likely the
case.
>
> Br,
> Petter
>
> 2017-12-07 21:39 GMT+01:00 David Alves <da...@gmail.com>:
>
>> Hi Petter
>>
>> I'd like to clarify what exactly happened and exactly what
>> are you
>> referring to as "inconsistency".
>> From what I understand of the first error you observed, the
>> Kudu was
>> underprovisioned, memory wise, and the ingest jobs/queries
>> failed. Is that
>> right? Since Kudu doesn't have atomic multi-row writes, it's
>> currently
>> expected in this case that you'll end up with partially written
>> data.
>> If you tried the same job again, and it succeeded, for
>> certain types of
>> operation (UPSERT, INSERT IGNORE) then the remaining rows would
>> be written
>> and all the data would be there as expected.
>> I'd like to distinguish this lack of atomicity on multi-row
>> transactions from "inconsistency", which is what you might
>> observe if an
>> operation didn't fail, but you couldn't see all the data. For
>> this latter
>> case there are options you can choose to avoid any
>> inconsistency.
>>
>> Best
>> David
>>
>>
>>
>> On Wed, Dec 6, 2017 at 4:26 AM, Petter von Dolwitz (Hem) <
>> petter.von.dolwitz@gmail.com> wrote:
>>
>>> Thanks for your reply Andrew!
>>>
>>> >How did you verify that all the data was inserted and how did
>>> >you find
>>> some data missing?
>>> This was done using Impala. We counted the rows for groups
>>> representing
>>> the chunks we inserted.
>>>
>>> >Following up on what I posted, take a look at
>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>> read_operations_scans. It seems definitely possible that not
>>> all of the
>>> rows had finished inserting when counting, or that the scans
>>> were sent to a
>>> stale replica.
>>> Before we shut down we could only see the following in the
>>> logs. I.e., no
>>> sign that ingestion was still ongoing.
>>>
>>> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
>>> 07:27:35.010694 90793 maintenance_manager.cc:383] P
>>> a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft
>>> memory limit
>>> (current capacity is 67.52%). However, there are no ops
>>> currently runnable
>>> which would free memory.
>>>
>>> Also the (cloudera) metric
>>> total_kudu_rows_inserted_rate_across_kudu_replicas
>>> showed zero.
>>>
>>> Still it seems like some data became inconsistent after
>>> restart. But if
>>> the maintenance_manager performs important jobs that are
>>> required to ensure
>>> that all data is inserted then I can understand why we ended
>>> up with
>>> inconsistent data. But, if I understand you correct, you are
>>> saying that
>>> these jobs are not critical for ingestion. In the link you
>>> provided I read
>>> "Impala scans are currently performed as READ_LATEST and have
>>> no
>>> consistency guarantees.". I would assume this means that it
>>> does not
>>> guarantee consistency if new data is inserted but should give
>>> valid (and
>>> same) results if no new data is inserted?
>>>
>>> I have not tried the ksck tool yet. Thank you for reminding. I
>>> will have
>>> a look.
>>>
>>> Br,
>>> Petter
>>>
>>>
>>> 2017-12-06 1:31 GMT+01:00 Andrew Wong <aw...@cloudera.com>:
>>>
>>>> How did you verify that all the data was inserted and how did
>>>> you find
>>>>> some data missing? I'm wondering if it's possible that the
>>>>> initial
>>>>> "missing" data was data that Kudu was still in the process
>>>>> of inserting
>>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>>
>>>>
>>>> Following up on what I posted, take a look at
>>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>>> read_operations_scans. It seems definitely possible that not
>>>> all of the
>>>> rows had finished inserting when counting, or that the scans
>>>> were sent to a
>>>> stale replica.
>>>>
>>>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong
>>>> <aw...@cloudera.com> wrote:
>>>>
>>>>> Hi Petter,
>>>>>
>>>>> When we verified that all data was inserted we found that
>>>>> some data was
>>>>>> missing. We added this missing data and on some chunks we
>>>>>> got the
>>>>>> information that all rows were already present, i.e impala
>>>>>> says something
>>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the
>>>>>> verification again now
>>>>>> shows that the Kudu table is complete. So, even though we
>>>>>> did not insert
>>>>>> any data on some chunks, a count(*) operation over these
>>>>>> chunks now returns
>>>>>> a different value.
>>>>>
>>>>>
>>>>> How did you verify that all the data was inserted and how
>>>>> did you find
>>>>> some data missing? I'm wondering if it's possible that the
>>>>> initial
>>>>> "missing" data was data that Kudu was still in the process
>>>>> of inserting
>>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>>
>>>>> Now to my question. Will data be inconsistent if we recycle
>>>>> Kudu after
>>>>>> seeing soft memory limit warnings?
>>>>>
>>>>>
>>>>> Your data should be consistently written, even with those
>>>>> warnings.
>>>>> AFAIK they would cause a bit of slowness, not incorrect
>>>>> results.
>>>>>
>>>>> Is there a way to tell when it is safe to restart Kudu to
>>>>> avoid these
>>>>>> issues? Should we use any special procedure when restarting
>>>>>> (e.g. only
>>>>>> restart the tablet servers, only restart one tablet server
>>>>>> at a time or
>>>>>> something like that)?
>>>>>
>>>>>
>>>>> In general, you can use the `ksck` tool to check the health
>>>>> of your
>>>>> cluster. See
>>>>> https://kudu.apache.org/docs/command_line_tools_referenc
>>>>> e.html#cluster-ksck for more details. For restarting a
>>>>> cluster, I
>>>>> would recommend taking down all tablet servers at once,
>>>>> otherwise tablet
>>>>> replicas may try to replicate data from the server that was
>>>>> taken down.
>>>>>
>>>>> Hope this helped,
>>>>> Andrew
>>>>>
>>>>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
>>>>> petter.von.dolwitz@gmail.com> wrote:
>>>>>
>>>>>> Hi Kudu users,
>>>>>>
>>>>>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a
>>>>>> baseline for
>>>>>> evaluation we ingested 3 month worth of data. During
>>>>>> ingestion we were
>>>>>> facing messages from the maintenance threads that a soft
>>>>>> memory limit were
>>>>>> reached. It seems like the background maintenance threads
>>>>>> stopped
>>>>>> performing their tasks at this point in time. It also so
>>>>>> seems like the
>>>>>> memory was never recovered even after stopping ingestion so
>>>>>> I guess there
>>>>>> was a large backlog being built up. I guess the root cause
>>>>>> here is that we
>>>>>> were a bit too conservative when giving Kudu memory. After
>>>>>> a reststart a
>>>>>> lot of maintenance tasks were started (i.e. compaction).
>>>>>>
>>>>>> When we verified that all data was inserted we found that
>>>>>> some data
>>>>>> was missing. We added this missing data and on some chunks
>>>>>> we got the
>>>>>> information that all rows were already present, i.e impala
>>>>>> says something
>>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the
>>>>>> verification again now
>>>>>> shows that the Kudu table is complete. So, even though we
>>>>>> did not insert
>>>>>> any data on some chunks, a count(*) operation over these
>>>>>> chunks now returns
>>>>>> a different value.
>>>>>>
>>>>>> Now to my question. Will data be inconsistent if we recycle
>>>>>> Kudu after
>>>>>> seeing soft memory limit warnings?
>>>>>>
>>>>>> Is there a way to tell when it is safe to restart Kudu to
>>>>>> avoid these
>>>>>> issues? Should we use any special procedure when restarting
>>>>>> (e.g. only
>>>>>> restart the tablet servers, only restart one tablet server
>>>>>> at a time or
>>>>>> something like that)?
>>>>>>
>>>>>> The table design uses 50 tablets per day (times 90
>>>>>> days). It is 8 TB
>>>>>> of data after 3xreplication over 5 tablet servers.
>>>>>>
>>>>>> Thanks,
>>>>>> Petter
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Andrew Wong
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Andrew Wong
>>>>
>>>
>>>
>>
--
David Alves
Re: Data inconsistency after restart
Posted by "Petter von Dolwitz (Hem)" <pe...@gmail.com>.
Hi David,
In short to summarize:
1. I ingest data. Kudus maintenance threads stops working (soft memory
limit) and incoming data is throttled. There are no errors reported on the
client side.
2. I stop ingestion and wait until i *think* Kudu is finsished.
3. I restart Kudu.
4. I validate the inserted data by doing count(*) on groups of data in
Kudu. For several groups, Kudu reports a lot of rows missing.
5. I ingest the same data again. Client reports that all row are already
present.
6. Doing the count(*) exercise again now gives me the correct number of
rows.
This tells me that the data was ingested into Kudu on the first attempt but
a scan did not find the data. Inserting the data again made it visible.
Br,
Petter
2017-12-07 21:39 GMT+01:00 David Alves <da...@gmail.com>:
> Hi Petter
>
> I'd like to clarify what exactly happened and exactly what are you
> referring to as "inconsistency".
> From what I understand of the first error you observed, the Kudu was
> underprovisioned, memory wise, and the ingest jobs/queries failed. Is that
> right? Since Kudu doesn't have atomic multi-row writes, it's currently
> expected in this case that you'll end up with partially written data.
> If you tried the same job again, and it succeeded, for certain types of
> operation (UPSERT, INSERT IGNORE) then the remaining rows would be written
> and all the data would be there as expected.
> I'd like to distinguish this lack of atomicity on multi-row
> transactions from "inconsistency", which is what you might observe if an
> operation didn't fail, but you couldn't see all the data. For this latter
> case there are options you can choose to avoid any inconsistency.
>
> Best
> David
>
>
>
> On Wed, Dec 6, 2017 at 4:26 AM, Petter von Dolwitz (Hem) <
> petter.von.dolwitz@gmail.com> wrote:
>
>> Thanks for your reply Andrew!
>>
>> >How did you verify that all the data was inserted and how did you find
>> some data missing?
>> This was done using Impala. We counted the rows for groups representing
>> the chunks we inserted.
>>
>> >Following up on what I posted, take a look at
>> https://kudu.apache.org/docs/transaction_semantics.html#_
>> read_operations_scans. It seems definitely possible that not all of the
>> rows had finished inserting when counting, or that the scans were sent to a
>> stale replica.
>> Before we shut down we could only see the following in the logs. I.e., no
>> sign that ingestion was still ongoing.
>>
>> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
>> 07:27:35.010694 90793 maintenance_manager.cc:383] P
>> a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft memory limit
>> (current capacity is 67.52%). However, there are no ops currently runnable
>> which would free memory.
>>
>> Also the (cloudera) metric total_kudu_rows_inserted_rate_across_kudu_replicas
>> showed zero.
>>
>> Still it seems like some data became inconsistent after restart. But if
>> the maintenance_manager performs important jobs that are required to ensure
>> that all data is inserted then I can understand why we ended up with
>> inconsistent data. But, if I understand you correct, you are saying that
>> these jobs are not critical for ingestion. In the link you provided I read
>> "Impala scans are currently performed as READ_LATEST and have no
>> consistency guarantees.". I would assume this means that it does not
>> guarantee consistency if new data is inserted but should give valid (and
>> same) results if no new data is inserted?
>>
>> I have not tried the ksck tool yet. Thank you for reminding. I will have
>> a look.
>>
>> Br,
>> Petter
>>
>>
>> 2017-12-06 1:31 GMT+01:00 Andrew Wong <aw...@cloudera.com>:
>>
>>> How did you verify that all the data was inserted and how did you find
>>>> some data missing? I'm wondering if it's possible that the initial
>>>> "missing" data was data that Kudu was still in the process of inserting
>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>
>>>
>>> Following up on what I posted, take a look at
>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>> read_operations_scans. It seems definitely possible that not all of the
>>> rows had finished inserting when counting, or that the scans were sent to a
>>> stale replica.
>>>
>>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong <aw...@cloudera.com> wrote:
>>>
>>>> Hi Petter,
>>>>
>>>> When we verified that all data was inserted we found that some data was
>>>>> missing. We added this missing data and on some chunks we got the
>>>>> information that all rows were already present, i.e impala says something
>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>>> a different value.
>>>>
>>>>
>>>> How did you verify that all the data was inserted and how did you find
>>>> some data missing? I'm wondering if it's possible that the initial
>>>> "missing" data was data that Kudu was still in the process of inserting
>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>
>>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>>>> seeing soft memory limit warnings?
>>>>
>>>>
>>>> Your data should be consistently written, even with those warnings.
>>>> AFAIK they would cause a bit of slowness, not incorrect results.
>>>>
>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>>> restart the tablet servers, only restart one tablet server at a time or
>>>>> something like that)?
>>>>
>>>>
>>>> In general, you can use the `ksck` tool to check the health of your
>>>> cluster. See https://kudu.apache.org/docs/command_line_tools_referenc
>>>> e.html#cluster-ksck for more details. For restarting a cluster, I
>>>> would recommend taking down all tablet servers at once, otherwise tablet
>>>> replicas may try to replicate data from the server that was taken down.
>>>>
>>>> Hope this helped,
>>>> Andrew
>>>>
>>>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
>>>> petter.von.dolwitz@gmail.com> wrote:
>>>>
>>>>> Hi Kudu users,
>>>>>
>>>>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline for
>>>>> evaluation we ingested 3 month worth of data. During ingestion we were
>>>>> facing messages from the maintenance threads that a soft memory limit were
>>>>> reached. It seems like the background maintenance threads stopped
>>>>> performing their tasks at this point in time. It also so seems like the
>>>>> memory was never recovered even after stopping ingestion so I guess there
>>>>> was a large backlog being built up. I guess the root cause here is that we
>>>>> were a bit too conservative when giving Kudu memory. After a reststart a
>>>>> lot of maintenance tasks were started (i.e. compaction).
>>>>>
>>>>> When we verified that all data was inserted we found that some data
>>>>> was missing. We added this missing data and on some chunks we got the
>>>>> information that all rows were already present, i.e impala says something
>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>>> a different value.
>>>>>
>>>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>>>> seeing soft memory limit warnings?
>>>>>
>>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>>> restart the tablet servers, only restart one tablet server at a time or
>>>>> something like that)?
>>>>>
>>>>> The table design uses 50 tablets per day (times 90 days). It is 8 TB
>>>>> of data after 3xreplication over 5 tablet servers.
>>>>>
>>>>> Thanks,
>>>>> Petter
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Andrew Wong
>>>>
>>>
>>>
>>>
>>> --
>>> Andrew Wong
>>>
>>
>>
>
Re: Data inconsistency after restart
Posted by David Alves <da...@gmail.com>.
Hi Petter
I'd like to clarify what exactly happened and exactly what are you
referring to as "inconsistency".
From what I understand of the first error you observed, the Kudu was
underprovisioned, memory wise, and the ingest jobs/queries failed. Is that
right? Since Kudu doesn't have atomic multi-row writes, it's currently
expected in this case that you'll end up with partially written data.
If you tried the same job again, and it succeeded, for certain types of
operation (UPSERT, INSERT IGNORE) then the remaining rows would be written
and all the data would be there as expected.
I'd like to distinguish this lack of atomicity on multi-row transactions
from "inconsistency", which is what you might observe if an operation
didn't fail, but you couldn't see all the data. For this latter case there
are options you can choose to avoid any inconsistency.
Best
David
On Wed, Dec 6, 2017 at 4:26 AM, Petter von Dolwitz (Hem) <
petter.von.dolwitz@gmail.com> wrote:
> Thanks for your reply Andrew!
>
> >How did you verify that all the data was inserted and how did you find
> some data missing?
> This was done using Impala. We counted the rows for groups representing
> the chunks we inserted.
>
> >Following up on what I posted, take a look at https://kudu.apache.org/doc
> s/transaction_semantics.html#_read_operations_scans. It seems definitely
> possible that not all of the rows had finished inserting when counting, or
> that the scans were sent to a stale replica.
> Before we shut down we could only see the following in the logs. I.e., no
> sign that ingestion was still ongoing.
>
> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
> 07:27:35.010694 90793 maintenance_manager.cc:383] P
> a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft memory limit
> (current capacity is 67.52%). However, there are no ops currently runnable
> which would free memory.
>
> Also the (cloudera) metric total_kudu_rows_inserted_rate_across_kudu_replicas
> showed zero.
>
> Still it seems like some data became inconsistent after restart. But if
> the maintenance_manager performs important jobs that are required to ensure
> that all data is inserted then I can understand why we ended up with
> inconsistent data. But, if I understand you correct, you are saying that
> these jobs are not critical for ingestion. In the link you provided I read
> "Impala scans are currently performed as READ_LATEST and have no
> consistency guarantees.". I would assume this means that it does not
> guarantee consistency if new data is inserted but should give valid (and
> same) results if no new data is inserted?
>
> I have not tried the ksck tool yet. Thank you for reminding. I will have a
> look.
>
> Br,
> Petter
>
>
> 2017-12-06 1:31 GMT+01:00 Andrew Wong <aw...@cloudera.com>:
>
>> How did you verify that all the data was inserted and how did you find
>>> some data missing? I'm wondering if it's possible that the initial
>>> "missing" data was data that Kudu was still in the process of inserting
>>> (albeit slowly, due to memory backpressure or somesuch).
>>>
>>
>> Following up on what I posted, take a look at https://kudu.apache.org/doc
>> s/transaction_semantics.html#_read_operations_scans. It seems definitely
>> possible that not all of the rows had finished inserting when counting, or
>> that the scans were sent to a stale replica.
>>
>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong <aw...@cloudera.com> wrote:
>>
>>> Hi Petter,
>>>
>>> When we verified that all data was inserted we found that some data was
>>>> missing. We added this missing data and on some chunks we got the
>>>> information that all rows were already present, i.e impala says something
>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>> a different value.
>>>
>>>
>>> How did you verify that all the data was inserted and how did you find
>>> some data missing? I'm wondering if it's possible that the initial
>>> "missing" data was data that Kudu was still in the process of inserting
>>> (albeit slowly, due to memory backpressure or somesuch).
>>>
>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>>> seeing soft memory limit warnings?
>>>
>>>
>>> Your data should be consistently written, even with those warnings.
>>> AFAIK they would cause a bit of slowness, not incorrect results.
>>>
>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>> restart the tablet servers, only restart one tablet server at a time or
>>>> something like that)?
>>>
>>>
>>> In general, you can use the `ksck` tool to check the health of your
>>> cluster. See https://kudu.apache.org/docs/command_line_tools_referenc
>>> e.html#cluster-ksck for more details. For restarting a cluster, I would
>>> recommend taking down all tablet servers at once, otherwise tablet replicas
>>> may try to replicate data from the server that was taken down.
>>>
>>> Hope this helped,
>>> Andrew
>>>
>>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
>>> petter.von.dolwitz@gmail.com> wrote:
>>>
>>>> Hi Kudu users,
>>>>
>>>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline for
>>>> evaluation we ingested 3 month worth of data. During ingestion we were
>>>> facing messages from the maintenance threads that a soft memory limit were
>>>> reached. It seems like the background maintenance threads stopped
>>>> performing their tasks at this point in time. It also so seems like the
>>>> memory was never recovered even after stopping ingestion so I guess there
>>>> was a large backlog being built up. I guess the root cause here is that we
>>>> were a bit too conservative when giving Kudu memory. After a reststart a
>>>> lot of maintenance tasks were started (i.e. compaction).
>>>>
>>>> When we verified that all data was inserted we found that some data was
>>>> missing. We added this missing data and on some chunks we got the
>>>> information that all rows were already present, i.e impala says something
>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>> a different value.
>>>>
>>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>>> seeing soft memory limit warnings?
>>>>
>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>> restart the tablet servers, only restart one tablet server at a time or
>>>> something like that)?
>>>>
>>>> The table design uses 50 tablets per day (times 90 days). It is 8 TB of
>>>> data after 3xreplication over 5 tablet servers.
>>>>
>>>> Thanks,
>>>> Petter
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Andrew Wong
>>>
>>
>>
>>
>> --
>> Andrew Wong
>>
>
>
Re: Data inconsistency after restart
Posted by David Alves <da...@gmail.com>.
(re-sending as apparently a previous version of this message got lost)
Hi Petter
I'd like to clarify what exactly happened and exactly what are you
referring to as "inconsistency".
From what I understand of the first error you observed, the Kudu
was underprovisioned, memory wise, and the ingest jobs/queries failed.
Is that right? Since Kudu doesn't have atomic multi-row writes, it's
currently expected in this case that you'll end up with partially
written data.
If you tried the same job again, and it succeeded, for certain
types of operation (UPSERT, INSERT IGNORE) then the remaining rows
would be written and all the data would be there as expected. Which
kind of operation are you using to insert the data?
I'd like to understand the precise context better. Particularly I'd
like to distinguish this lack of atomicity on multi-row transactions
from "inconsistency" (in the CAP sense), which is what you might
observe if an operation didn't fail, but you couldn't see all the
data. For this latter case there are options you can choose, and we
can go through those, if that's the case.
Best
On Thu, Dec 7, 2017 at 12:34 PM, Petter von Dolwitz (Hem)
<pe...@gmail.com> wrote:
> Hi Andrew and Alexey,
>
> thanks for taking the time to answer.
>
>>Interesting. Just to be sure, was that seen on one tserver, or did you see
>> them across all of them?
> I did not look at all tservers at the time but it was one or two out of five
> so this could be a reason if one of more tservers had remaining work. The
> inserted_rows metric was steady at zero though.
>
>>Did you verified the client didn't report any errors during data ingestion?
>> Most likely you did, but I just wanted to make sure. BTW, what kind of
>> client did you use for the data ingestion?
> I don't remember seeing any errors at the client side. I used 4 parallell
> impala jobs as clients. When memory was not constrained I could see
> ingestion rates steady around 1.5 million rows / second. After a while
> (couple of hours perhaps), the ingestion rate dropped significantly so I
> decided to stop ingesting and restart kudu with more memory (after the logs
> stopped spinning at the tserver(s) I was looking at). I guess this was due
> to a backlog building up and that kudu throttled the incoming traffic.
>
> I could try to repeat the exercise and try to record result more
> scientifically.
>
> Br,
> Petter
>
>
>
>
> 2017-12-07 20:48 GMT+01:00 Alexey Serbin <as...@cloudera.com>:
>>
>> Hi Petter,
>>
>> Before going too deep in attempts to find the place where the data was
>> lost, I just wanted to make sure we definitely know that the data was
>> delivered from the client to the server side.
>>
>> Did you verified the client didn't report any errors during data
>> ingestion? Most likely you did, but I just wanted to make sure. BTW, what
>> kind of client did you use for the data ingestion?
>>
>> Thanks!
>>
>>
>> Kind regards,
>>
>> Alexey
>>
>>
>> On 12/6/17 3:56 PM, Andrew Wong wrote:
>>>
>>> Hi Petter,
>>>
>>> Before we shut down we could only see the following in the logs.
>>> I.e., no sign that ingestion was still ongoing.
>>>
>>>
>>> Interesting. Just to be sure, was that seen on one tserver, or did you
>>> see them across all of them?
>>>
>>> But if the maintenance_manager performs important jobs that are
>>> required to ensure that all data is inserted then I can understand
>>> why we ended up with inconsistent data.
>>>
>>>
>>> The maintenance manager's role is somewhat orthogonal to writes: data is
>>> first written to the on-disk write-ahead log and also kept in-memory to be
>>> accessible by scans. The maintenance manager periodically shuttles this
>>> in-memory data to disk, among various other tasks like cleaning up WAL
>>> segments, compacting rowsets, etc. Given that, a lack of maintenance ops
>>> shouldn't cause incorrectness in data, even after restarting.
>>>
>>> I would assume this means that it does not guarantee consistency
>>> if new data is inserted but should give valid (and same) results
>>> if no new data is inserted?
>>>
>>>
>>> Right, if /all/ tservers a truly caught up and done processing the
>>> writes, with no tablet copies going on, and with no new data coming in, then
>>> the results should be consistent.
>>>
>>>
>>> Hope this helped,
>>> Andrew
>>>
>>> On Wed, Dec 6, 2017 at 7:33 AM, Boris Tyukin <boris@boristyukin.com
>>> <ma...@boristyukin.com>> wrote:
>>>
>>> this is smart, we are doing the same thing but the best part that
>>> attracts me to Kudu is replacing our main HDFS storage with Kudu
>>> to enable near RT use cases and not to deal with HBase and a
>>> Lambda architecture mess so reliability and scalability is a big
>>> deal for us as we are looking to move most of our data to Kudu.
>>>
>>> On Wed, Dec 6, 2017 at 9:58 AM, Petter von Dolwitz (Hem)
>>> <petter.von.dolwitz@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Hi Boris,
>>>
>>> we do not have a Cloudera contract at the moment. Until we
>>> gained more Kudu experience we keep our master data in parquet
>>> format so that we can rebuild Kudu-tables upon errors. We are
>>> still in the early learning phase.
>>>
>>> Br,
>>> Petter
>>>
>>>
>>>
>>> 2017-12-06 14:35 GMT+01:00 Boris Tyukin <boris@boristyukin.com
>>> <ma...@boristyukin.com>>:
>>>
>>> this is definitely concerning thread for us looking to use
>>> Impala for storing mission-critical company data. Petter,
>>> are you paid Cloudera customer btw? I wonder if you opened
>>> support ticket as well
>>>
>>> On Wed, Dec 6, 2017 at 7:26 AM, Petter von Dolwitz (Hem)
>>> <petter.von.dolwitz@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Thanks for your reply Andrew!
>>>
>>> >How did you verify that all the data was inserted and
>>> how did you find some data missing?
>>> This was done using Impala. We counted the rows for
>>> groups representing the chunks we inserted.
>>>
>>> >Following up on what I posted, take a look at
>>>
>>> https://kudu.apache.org/docs/transaction_semantics.html#_read_operations_scans
>>>
>>> <https://kudu.apache.org/docs/transaction_semantics.html#_read_operations_scans>.
>>> It seems definitely possible that not all of the rows
>>> had finished inserting when counting, or that the
>>> scans were sent to a stale replica.
>>> Before we shut down we could only see the following in
>>> the logs. I.e., no sign that ingestion was still ongoing.
>>>
>>>
>>> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
>>> 07:27:35.010694 90793 maintenance_manager.cc:383] P
>>> a38902afefca4a85a5469d149df9b4cb: we have exceeded our
>>> soft memory limit (current capacity is 67.52%).
>>> However, there are no ops currently runnable which
>>> would free memory.
>>>
>>> Also the (cloudera) metric
>>> total_kudu_rows_inserted_rate_across_kudu_replicas
>>> showed zero.
>>>
>>> Still it seems like some data became inconsistent
>>> after restart. But if the maintenance_manager performs
>>> important jobs that are required to ensure that all
>>> data is inserted then I can understand why we ended up
>>> with inconsistent data. But, if I understand you
>>> correct, you are saying that these jobs are not
>>> critical for ingestion. In the link you provided I
>>> read "Impala scans are currently performed as
>>> READ_LATEST and have no consistency guarantees.". I
>>> would assume this means that it does not guarantee
>>> consistency if new data is inserted but should give
>>> valid (and same) results if no new data is inserted?
>>>
>>> I have not tried the ksck tool yet. Thank you for
>>> reminding. I will have a look.
>>>
>>> Br,
>>> Petter
>>>
>>>
>>> 2017-12-06 1:31 GMT+01:00 Andrew Wong
>>> <awong@cloudera.com <ma...@cloudera.com>>:
>>>
>>> How did you verify that all the data was
>>> inserted and how did you find some data
>>> missing? I'm wondering if it's possible that
>>> the initial "missing" data was data that Kudu
>>> was still in the process of inserting (albeit
>>> slowly, due to memory backpressure or somesuch).
>>>
>>>
>>> Following up on what I posted, take a look at
>>>
>>> https://kudu.apache.org/docs/transaction_semantics.html#_read_operations_scans
>>>
>>> <https://kudu.apache.org/docs/transaction_semantics.html#_read_operations_scans>.
>>> It seems definitely possible that not all of the
>>> rows had finished inserting when counting, or that
>>> the scans were sent to a stale replica.
>>>
>>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong
>>> <awong@cloudera.com <ma...@cloudera.com>>
>>>
>>> wrote:
>>>
>>> Hi Petter,
>>>
>>> When we verified that all data was
>>> inserted we found that some data was
>>> missing. We added this missing data and on
>>> some chunks we got the information that
>>> all rows were already present, i.e impala
>>> says something like Modified: 0 rows,
>>> nnnnnnn errors. Doing the verification
>>> again now shows that the Kudu table is
>>> complete. So, even though we did not
>>> insert any data on some chunks, a count(*)
>>> operation over these chunks now returns a
>>> different value.
>>>
>>>
>>> How did you verify that all the data was
>>> inserted and how did you find some data
>>> missing? I'm wondering if it's possible that
>>> the initial "missing" data was data that Kudu
>>> was still in the process of inserting (albeit
>>> slowly, due to memory backpressure or somesuch).
>>>
>>> Now to my question. Will data be
>>> inconsistent if we recycle Kudu after
>>> seeing soft memory limit warnings?
>>>
>>>
>>> Your data should be consistently written, even
>>> with those warnings. AFAIK they would cause a
>>> bit of slowness, not incorrect results.
>>>
>>> Is there a way to tell when it is safe to
>>> restart Kudu to avoid these issues? Should
>>> we use any special procedure when
>>> restarting (e.g. only restart the tablet
>>> servers, only restart one tablet server at
>>> a time or something like that)?
>>>
>>>
>>> In general, you can use the `ksck` tool to
>>> check the health of your cluster. See
>>>
>>> https://kudu.apache.org/docs/command_line_tools_reference.html#cluster-ksck
>>>
>>> <https://kudu.apache.org/docs/command_line_tools_reference.html#cluster-ksck>
>>> for more details. For restarting a cluster, I
>>> would recommend taking down all tablet servers
>>> at once, otherwise tablet replicas may try to
>>> replicate data from the server that was taken
>>> down.
>>>
>>> Hope this helped,
>>> Andrew
>>>
>>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von
>>> Dolwitz (Hem) <petter.von.dolwitz@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Hi Kudu users,
>>>
>>> We just started to use Kudu
>>> (1.4.0+cdh5.12.1). To make a baseline for
>>> evaluation we ingested 3 month worth of
>>> data. During ingestion we were facing
>>> messages from the maintenance threads that
>>> a soft memory limit were reached. It seems
>>> like the background maintenance threads
>>> stopped performing their tasks at this
>>> point in time. It also so seems like the
>>> memory was never recovered even after
>>> stopping ingestion so I guess there was a
>>> large backlog being built up. I guess the
>>> root cause here is that we were a bit too
>>> conservative when giving Kudu memory.
>>> After a reststart a lot of maintenance
>>> tasks were started (i.e. compaction).
>>>
>>> When we verified that all data was
>>> inserted we found that some data was
>>> missing. We added this missing data and on
>>> some chunks we got the information that
>>> all rows were already present, i.e impala
>>> says something like Modified: 0 rows,
>>> nnnnnnn errors. Doing the verification
>>> again now shows that the Kudu table is
>>> complete. So, even though we did not
>>> insert any data on some chunks, a count(*)
>>> operation over these chunks now returns a
>>> different value.
>>>
>>> Now to my question. Will data be
>>> inconsistent if we recycle Kudu after
>>> seeing soft memory limit warnings?
>>>
>>> Is there a way to tell when it is safe to
>>> restart Kudu to avoid these issues? Should
>>> we use any special procedure when
>>> restarting (e.g. only restart the tablet
>>> servers, only restart one tablet server at
>>> a time or something like that)?
>>>
>>> The table design uses 50 tablets per day
>>> (times 90 days). It is 8 TB of data after
>>> 3xreplication over 5 tablet servers.
>>>
>>> Thanks,
>>> Petter
>>>
>>>
>>>
>>>
>>>
>>> -- Andrew Wong
>>>
>>>
>>>
>>>
>>> -- Andrew Wong
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Andrew Wong
>>
>>
>
Re: Data inconsistency after restart
Posted by "Petter von Dolwitz (Hem)" <pe...@gmail.com>.
Hi Andrew and Alexey,
thanks for taking the time to answer.
>Interesting. Just to be sure, was that seen on one tserver, or did you see
them across all of them?
I did not look at all tservers at the time but it was one or two out of
five so this could be a reason if one of more tservers had remaining work.
The inserted_rows metric was steady at zero though.
>Did you verified the client didn't report any errors during data
ingestion? Most likely you did, but I just wanted to make sure. BTW, what
kind of client did you use for the data ingestion?
I don't remember seeing any errors at the client side. I used 4 parallell
impala jobs as clients. When memory was not constrained I could see
ingestion rates steady around 1.5 million rows / second. After a while
(couple of hours perhaps), the ingestion rate dropped significantly so I
decided to stop ingesting and restart kudu with more memory (after the logs
stopped spinning at the tserver(s) I was looking at). I guess this was due
to a backlog building up and that kudu throttled the incoming traffic.
I could try to repeat the exercise and try to record result more
scientifically.
Br,
Petter
2017-12-07 20:48 GMT+01:00 Alexey Serbin <as...@cloudera.com>:
> Hi Petter,
>
> Before going too deep in attempts to find the place where the data was
> lost, I just wanted to make sure we definitely know that the data was
> delivered from the client to the server side.
>
> Did you verified the client didn't report any errors during data
> ingestion? Most likely you did, but I just wanted to make sure. BTW, what
> kind of client did you use for the data ingestion?
>
> Thanks!
>
>
> Kind regards,
>
> Alexey
>
>
> On 12/6/17 3:56 PM, Andrew Wong wrote:
>
>> Hi Petter,
>>
>> Before we shut down we could only see the following in the logs.
>> I.e., no sign that ingestion was still ongoing.
>>
>>
>> Interesting. Just to be sure, was that seen on one tserver, or did you
>> see them across all of them?
>>
>> But if the maintenance_manager performs important jobs that are
>> required to ensure that all data is inserted then I can understand
>> why we ended up with inconsistent data.
>>
>>
>> The maintenance manager's role is somewhat orthogonal to writes: data is
>> first written to the on-disk write-ahead log and also kept in-memory to be
>> accessible by scans. The maintenance manager periodically shuttles this
>> in-memory data to disk, among various other tasks like cleaning up WAL
>> segments, compacting rowsets, etc. Given that, a lack of maintenance ops
>> shouldn't cause incorrectness in data, even after restarting.
>>
>> I would assume this means that it does not guarantee consistency
>> if new data is inserted but should give valid (and same) results
>> if no new data is inserted?
>>
>>
>> Right, if /all/ tservers a truly caught up and done processing the
>> writes, with no tablet copies going on, and with no new data coming in,
>> then the results should be consistent.
>>
>>
>> Hope this helped,
>> Andrew
>>
>> On Wed, Dec 6, 2017 at 7:33 AM, Boris Tyukin <boris@boristyukin.com
>> <ma...@boristyukin.com>> wrote:
>>
>> this is smart, we are doing the same thing but the best part that
>> attracts me to Kudu is replacing our main HDFS storage with Kudu
>> to enable near RT use cases and not to deal with HBase and a
>> Lambda architecture mess so reliability and scalability is a big
>> deal for us as we are looking to move most of our data to Kudu.
>>
>> On Wed, Dec 6, 2017 at 9:58 AM, Petter von Dolwitz (Hem)
>> <petter.von.dolwitz@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hi Boris,
>>
>> we do not have a Cloudera contract at the moment. Until we
>> gained more Kudu experience we keep our master data in parquet
>> format so that we can rebuild Kudu-tables upon errors. We are
>> still in the early learning phase.
>>
>> Br,
>> Petter
>>
>>
>>
>> 2017-12-06 14:35 GMT+01:00 Boris Tyukin <boris@boristyukin.com
>> <ma...@boristyukin.com>>:
>>
>> this is definitely concerning thread for us looking to use
>> Impala for storing mission-critical company data. Petter,
>> are you paid Cloudera customer btw? I wonder if you opened
>> support ticket as well
>>
>> On Wed, Dec 6, 2017 at 7:26 AM, Petter von Dolwitz (Hem)
>> <petter.von.dolwitz@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Thanks for your reply Andrew!
>>
>> >How did you verify that all the data was inserted and
>> how did you find some data missing?
>> This was done using Impala. We counted the rows for
>> groups representing the chunks we inserted.
>>
>> >Following up on what I posted, take a look at
>> https://kudu.apache.org/docs/t
>> ransaction_semantics.html#_read_operations_scans
>> <https://kudu.apache.org/docs/
>> transaction_semantics.html#_read_operations_scans>.
>> It seems definitely possible that not all of the rows
>> had finished inserting when counting, or that the
>> scans were sent to a stale replica.
>> Before we shut down we could only see the following in
>> the logs. I.e., no sign that ingestion was still ongoing.
>>
>> kudu-tserver.ip-xx-yyy-z-nnn.r
>> oot.log.INFO.20171201-065232.90314:I1201
>> 07:27:35.010694 90793 maintenance_manager.cc:383] P
>> a38902afefca4a85a5469d149df9b4cb: we have exceeded our
>> soft memory limit (current capacity is 67.52%).
>> However, there are no ops currently runnable which
>> would free memory.
>>
>> Also the (cloudera) metric
>> total_kudu_rows_inserted_rate_across_kudu_replicas
>> showed zero.
>>
>> Still it seems like some data became inconsistent
>> after restart. But if the maintenance_manager performs
>> important jobs that are required to ensure that all
>> data is inserted then I can understand why we ended up
>> with inconsistent data. But, if I understand you
>> correct, you are saying that these jobs are not
>> critical for ingestion. In the link you provided I
>> read "Impala scans are currently performed as
>> READ_LATEST and have no consistency guarantees.". I
>> would assume this means that it does not guarantee
>> consistency if new data is inserted but should give
>> valid (and same) results if no new data is inserted?
>>
>> I have not tried the ksck tool yet. Thank you for
>> reminding. I will have a look.
>>
>> Br,
>> Petter
>>
>>
>> 2017-12-06 1:31 GMT+01:00 Andrew Wong
>> <awong@cloudera.com <ma...@cloudera.com>>:
>>
>> How did you verify that all the data was
>> inserted and how did you find some data
>> missing? I'm wondering if it's possible that
>> the initial "missing" data was data that Kudu
>> was still in the process of inserting (albeit
>> slowly, due to memory backpressure or somesuch).
>>
>>
>> Following up on what I posted, take a look at
>> https://kudu.apache.org/docs/t
>> ransaction_semantics.html#_read_operations_scans
>> <https://kudu.apache.org/docs/
>> transaction_semantics.html#_read_operations_scans>.
>> It seems definitely possible that not all of the
>> rows had finished inserting when counting, or that
>> the scans were sent to a stale replica.
>>
>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong
>> <awong@cloudera.com <ma...@cloudera.com>>
>>
>> wrote:
>>
>> Hi Petter,
>>
>> When we verified that all data was
>> inserted we found that some data was
>> missing. We added this missing data and on
>> some chunks we got the information that
>> all rows were already present, i.e impala
>> says something like Modified: 0 rows,
>> nnnnnnn errors. Doing the verification
>> again now shows that the Kudu table is
>> complete. So, even though we did not
>> insert any data on some chunks, a count(*)
>> operation over these chunks now returns a
>> different value.
>>
>>
>> How did you verify that all the data was
>> inserted and how did you find some data
>> missing? I'm wondering if it's possible that
>> the initial "missing" data was data that Kudu
>> was still in the process of inserting (albeit
>> slowly, due to memory backpressure or somesuch).
>>
>> Now to my question. Will data be
>> inconsistent if we recycle Kudu after
>> seeing soft memory limit warnings?
>>
>>
>> Your data should be consistently written, even
>> with those warnings. AFAIK they would cause a
>> bit of slowness, not incorrect results.
>>
>> Is there a way to tell when it is safe to
>> restart Kudu to avoid these issues? Should
>> we use any special procedure when
>> restarting (e.g. only restart the tablet
>> servers, only restart one tablet server at
>> a time or something like that)?
>>
>>
>> In general, you can use the `ksck` tool to
>> check the health of your cluster. See
>> https://kudu.apache.org/docs/c
>> ommand_line_tools_reference.html#cluster-ksck
>> <https://kudu.apache.org/docs/
>> command_line_tools_reference.html#cluster-ksck>
>> for more details. For restarting a cluster, I
>> would recommend taking down all tablet servers
>> at once, otherwise tablet replicas may try to
>> replicate data from the server that was taken
>> down.
>>
>> Hope this helped,
>> Andrew
>>
>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von
>> Dolwitz (Hem) <petter.von.dolwitz@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hi Kudu users,
>>
>> We just started to use Kudu
>> (1.4.0+cdh5.12.1). To make a baseline for
>> evaluation we ingested 3 month worth of
>> data. During ingestion we were facing
>> messages from the maintenance threads that
>> a soft memory limit were reached. It seems
>> like the background maintenance threads
>> stopped performing their tasks at this
>> point in time. It also so seems like the
>> memory was never recovered even after
>> stopping ingestion so I guess there was a
>> large backlog being built up. I guess the
>> root cause here is that we were a bit too
>> conservative when giving Kudu memory.
>> After a reststart a lot of maintenance
>> tasks were started (i.e. compaction).
>>
>> When we verified that all data was
>> inserted we found that some data was
>> missing. We added this missing data and on
>> some chunks we got the information that
>> all rows were already present, i.e impala
>> says something like Modified: 0 rows,
>> nnnnnnn errors. Doing the verification
>> again now shows that the Kudu table is
>> complete. So, even though we did not
>> insert any data on some chunks, a count(*)
>> operation over these chunks now returns a
>> different value.
>>
>> Now to my question. Will data be
>> inconsistent if we recycle Kudu after
>> seeing soft memory limit warnings?
>>
>> Is there a way to tell when it is safe to
>> restart Kudu to avoid these issues? Should
>> we use any special procedure when
>> restarting (e.g. only restart the tablet
>> servers, only restart one tablet server at
>> a time or something like that)?
>>
>> The table design uses 50 tablets per day
>> (times 90 days). It is 8 TB of data after
>> 3xreplication over 5 tablet servers.
>>
>> Thanks,
>> Petter
>>
>>
>>
>>
>>
>> -- Andrew Wong
>>
>>
>>
>>
>> -- Andrew Wong
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Andrew Wong
>>
>
>
Re: Data inconsistency after restart
Posted by Alexey Serbin <as...@cloudera.com>.
Hi Petter,
Before going too deep in attempts to find the place where the data was
lost, I just wanted to make sure we definitely know that the data was
delivered from the client to the server side.
Did you verified the client didn't report any errors during data
ingestion? Most likely you did, but I just wanted to make sure. BTW,
what kind of client did you use for the data ingestion?
Thanks!
Kind regards,
Alexey
On 12/6/17 3:56 PM, Andrew Wong wrote:
> Hi Petter,
>
> Before we shut down we could only see the following in the logs.
> I.e., no sign that ingestion was still ongoing.
>
>
> Interesting. Just to be sure, was that seen on one tserver, or did you
> see them across all of them?
>
> But if the maintenance_manager performs important jobs that are
> required to ensure that all data is inserted then I can understand
> why we ended up with inconsistent data.
>
>
> The maintenance manager's role is somewhat orthogonal to writes: data
> is first written to the on-disk write-ahead log and also kept
> in-memory to be accessible by scans. The maintenance manager
> periodically shuttles this in-memory data to disk, among various other
> tasks like cleaning up WAL segments, compacting rowsets, etc. Given
> that, a lack of maintenance ops shouldn't cause incorrectness in data,
> even after restarting.
>
> I would assume this means that it does not guarantee consistency
> if new data is inserted but should give valid (and same) results
> if no new data is inserted?
>
>
> Right, if /all/ tservers a truly caught up and done processing the
> writes, with no tablet copies going on, and with no new data coming
> in, then the results should be consistent.
>
>
> Hope this helped,
> Andrew
>
> On Wed, Dec 6, 2017 at 7:33 AM, Boris Tyukin <boris@boristyukin.com
> <ma...@boristyukin.com>> wrote:
>
> this is smart, we are doing the same thing but the best part that
> attracts me to Kudu is replacing our main HDFS storage with Kudu
> to enable near RT use cases and not to deal with HBase and a
> Lambda architecture mess so reliability and scalability is a big
> deal for us as we are looking to move most of our data to Kudu.
>
> On Wed, Dec 6, 2017 at 9:58 AM, Petter von Dolwitz (Hem)
> <petter.von.dolwitz@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi Boris,
>
> we do not have a Cloudera contract at the moment. Until we
> gained more Kudu experience we keep our master data in parquet
> format so that we can rebuild Kudu-tables upon errors. We are
> still in the early learning phase.
>
> Br,
> Petter
>
>
>
> 2017-12-06 14:35 GMT+01:00 Boris Tyukin <boris@boristyukin.com
> <ma...@boristyukin.com>>:
>
> this is definitely concerning thread for us looking to use
> Impala for storing mission-critical company data. Petter,
> are you paid Cloudera customer btw? I wonder if you opened
> support ticket as well
>
> On Wed, Dec 6, 2017 at 7:26 AM, Petter von Dolwitz (Hem)
> <petter.von.dolwitz@gmail.com
> <ma...@gmail.com>> wrote:
>
> Thanks for your reply Andrew!
>
> >How did you verify that all the data was inserted and
> how did you find some data missing?
> This was done using Impala. We counted the rows for
> groups representing the chunks we inserted.
>
> >Following up on what I posted, take a look at
> https://kudu.apache.org/docs/transaction_semantics.html#_read_operations_scans
> <https://kudu.apache.org/docs/transaction_semantics.html#_read_operations_scans>.
> It seems definitely possible that not all of the rows
> had finished inserting when counting, or that the
> scans were sent to a stale replica.
> Before we shut down we could only see the following in
> the logs. I.e., no sign that ingestion was still ongoing.
>
> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
> 07:27:35.010694 90793 maintenance_manager.cc:383] P
> a38902afefca4a85a5469d149df9b4cb: we have exceeded our
> soft memory limit (current capacity is 67.52%).
> However, there are no ops currently runnable which
> would free memory.
>
> Also the (cloudera) metric
> total_kudu_rows_inserted_rate_across_kudu_replicas
> showed zero.
>
> Still it seems like some data became inconsistent
> after restart. But if the maintenance_manager performs
> important jobs that are required to ensure that all
> data is inserted then I can understand why we ended up
> with inconsistent data. But, if I understand you
> correct, you are saying that these jobs are not
> critical for ingestion. In the link you provided I
> read "Impala scans are currently performed as
> READ_LATEST and have no consistency guarantees.". I
> would assume this means that it does not guarantee
> consistency if new data is inserted but should give
> valid (and same) results if no new data is inserted?
>
> I have not tried the ksck tool yet. Thank you for
> reminding. I will have a look.
>
> Br,
> Petter
>
>
> 2017-12-06 1:31 GMT+01:00 Andrew Wong
> <awong@cloudera.com <ma...@cloudera.com>>:
>
> How did you verify that all the data was
> inserted and how did you find some data
> missing? I'm wondering if it's possible that
> the initial "missing" data was data that Kudu
> was still in the process of inserting (albeit
> slowly, due to memory backpressure or somesuch).
>
>
> Following up on what I posted, take a look at
> https://kudu.apache.org/docs/transaction_semantics.html#_read_operations_scans
> <https://kudu.apache.org/docs/transaction_semantics.html#_read_operations_scans>.
> It seems definitely possible that not all of the
> rows had finished inserting when counting, or that
> the scans were sent to a stale replica.
>
> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong
> <awong@cloudera.com <ma...@cloudera.com>>
> wrote:
>
> Hi Petter,
>
> When we verified that all data was
> inserted we found that some data was
> missing. We added this missing data and on
> some chunks we got the information that
> all rows were already present, i.e impala
> says something like Modified: 0 rows,
> nnnnnnn errors. Doing the verification
> again now shows that the Kudu table is
> complete. So, even though we did not
> insert any data on some chunks, a count(*)
> operation over these chunks now returns a
> different value.
>
>
> How did you verify that all the data was
> inserted and how did you find some data
> missing? I'm wondering if it's possible that
> the initial "missing" data was data that Kudu
> was still in the process of inserting (albeit
> slowly, due to memory backpressure or somesuch).
>
> Now to my question. Will data be
> inconsistent if we recycle Kudu after
> seeing soft memory limit warnings?
>
>
> Your data should be consistently written, even
> with those warnings. AFAIK they would cause a
> bit of slowness, not incorrect results.
>
> Is there a way to tell when it is safe to
> restart Kudu to avoid these issues? Should
> we use any special procedure when
> restarting (e.g. only restart the tablet
> servers, only restart one tablet server at
> a time or something like that)?
>
>
> In general, you can use the `ksck` tool to
> check the health of your cluster. See
> https://kudu.apache.org/docs/command_line_tools_reference.html#cluster-ksck
> <https://kudu.apache.org/docs/command_line_tools_reference.html#cluster-ksck>
> for more details. For restarting a cluster, I
> would recommend taking down all tablet servers
> at once, otherwise tablet replicas may try to
> replicate data from the server that was taken
> down.
>
> Hope this helped,
> Andrew
>
> On Tue, Dec 5, 2017 at 10:42 AM, Petter von
> Dolwitz (Hem) <petter.von.dolwitz@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi Kudu users,
>
> We just started to use Kudu
> (1.4.0+cdh5.12.1). To make a baseline for
> evaluation we ingested 3 month worth of
> data. During ingestion we were facing
> messages from the maintenance threads that
> a soft memory limit were reached. It seems
> like the background maintenance threads
> stopped performing their tasks at this
> point in time. It also so seems like the
> memory was never recovered even after
> stopping ingestion so I guess there was a
> large backlog being built up. I guess the
> root cause here is that we were a bit too
> conservative when giving Kudu memory.
> After a reststart a lot of maintenance
> tasks were started (i.e. compaction).
>
> When we verified that all data was
> inserted we found that some data was
> missing. We added this missing data and on
> some chunks we got the information that
> all rows were already present, i.e impala
> says something like Modified: 0 rows,
> nnnnnnn errors. Doing the verification
> again now shows that the Kudu table is
> complete. So, even though we did not
> insert any data on some chunks, a count(*)
> operation over these chunks now returns a
> different value.
>
> Now to my question. Will data be
> inconsistent if we recycle Kudu after
> seeing soft memory limit warnings?
>
> Is there a way to tell when it is safe to
> restart Kudu to avoid these issues? Should
> we use any special procedure when
> restarting (e.g. only restart the tablet
> servers, only restart one tablet server at
> a time or something like that)?
>
> The table design uses 50 tablets per day
> (times 90 days). It is 8 TB of data after
> 3xreplication over 5 tablet servers.
>
> Thanks,
> Petter
>
>
>
>
>
> --
> Andrew Wong
>
>
>
>
> --
> Andrew Wong
>
>
>
>
>
>
>
>
> --
> Andrew Wong
Re: Data inconsistency after restart
Posted by Andrew Wong <aw...@cloudera.com>.
Hi Petter,
> Before we shut down we could only see the following in the logs. I.e., no
> sign that ingestion was still ongoing.
>
Interesting. Just to be sure, was that seen on one tserver, or did you see
them across all of them?
But if the maintenance_manager performs important jobs that are required to
> ensure that all data is inserted then I can understand why we ended up with
> inconsistent data.
The maintenance manager's role is somewhat orthogonal to writes: data is
first written to the on-disk write-ahead log and also kept in-memory to be
accessible by scans. The maintenance manager periodically shuttles this
in-memory data to disk, among various other tasks like cleaning up WAL
segments, compacting rowsets, etc. Given that, a lack of maintenance ops
shouldn't cause incorrectness in data, even after restarting.
I would assume this means that it does not guarantee consistency if new
> data is inserted but should give valid (and same) results if no new data is
> inserted?
Right, if *all* tservers a truly caught up and done processing the writes,
with no tablet copies going on, and with no new data coming in, then the
results should be consistent.
Hope this helped,
Andrew
On Wed, Dec 6, 2017 at 7:33 AM, Boris Tyukin <bo...@boristyukin.com> wrote:
> this is smart, we are doing the same thing but the best part that attracts
> me to Kudu is replacing our main HDFS storage with Kudu to enable near RT
> use cases and not to deal with HBase and a Lambda architecture mess so
> reliability and scalability is a big deal for us as we are looking to move
> most of our data to Kudu.
>
> On Wed, Dec 6, 2017 at 9:58 AM, Petter von Dolwitz (Hem) <
> petter.von.dolwitz@gmail.com> wrote:
>
>> Hi Boris,
>>
>> we do not have a Cloudera contract at the moment. Until we gained more
>> Kudu experience we keep our master data in parquet format so that we can
>> rebuild Kudu-tables upon errors. We are still in the early learning phase.
>>
>> Br,
>> Petter
>>
>>
>>
>> 2017-12-06 14:35 GMT+01:00 Boris Tyukin <bo...@boristyukin.com>:
>>
>>> this is definitely concerning thread for us looking to use Impala for
>>> storing mission-critical company data. Petter, are you paid Cloudera
>>> customer btw? I wonder if you opened support ticket as well
>>>
>>> On Wed, Dec 6, 2017 at 7:26 AM, Petter von Dolwitz (Hem) <
>>> petter.von.dolwitz@gmail.com> wrote:
>>>
>>>> Thanks for your reply Andrew!
>>>>
>>>> >How did you verify that all the data was inserted and how did you find
>>>> some data missing?
>>>> This was done using Impala. We counted the rows for groups representing
>>>> the chunks we inserted.
>>>>
>>>> >Following up on what I posted, take a look at
>>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>>> read_operations_scans. It seems definitely possible that not all of
>>>> the rows had finished inserting when counting, or that the scans were sent
>>>> to a stale replica.
>>>> Before we shut down we could only see the following in the logs. I.e.,
>>>> no sign that ingestion was still ongoing.
>>>>
>>>> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
>>>> 07:27:35.010694 90793 maintenance_manager.cc:383] P
>>>> a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft memory
>>>> limit (current capacity is 67.52%). However, there are no ops currently
>>>> runnable which would free memory.
>>>>
>>>> Also the (cloudera) metric total_kudu_rows_inserted_rate_across_kudu_replicas
>>>> showed zero.
>>>>
>>>> Still it seems like some data became inconsistent after restart. But if
>>>> the maintenance_manager performs important jobs that are required to ensure
>>>> that all data is inserted then I can understand why we ended up with
>>>> inconsistent data. But, if I understand you correct, you are saying that
>>>> these jobs are not critical for ingestion. In the link you provided I read
>>>> "Impala scans are currently performed as READ_LATEST and have no
>>>> consistency guarantees.". I would assume this means that it does not
>>>> guarantee consistency if new data is inserted but should give valid (and
>>>> same) results if no new data is inserted?
>>>>
>>>> I have not tried the ksck tool yet. Thank you for reminding. I will
>>>> have a look.
>>>>
>>>> Br,
>>>> Petter
>>>>
>>>>
>>>> 2017-12-06 1:31 GMT+01:00 Andrew Wong <aw...@cloudera.com>:
>>>>
>>>>> How did you verify that all the data was inserted and how did you find
>>>>>> some data missing? I'm wondering if it's possible that the initial
>>>>>> "missing" data was data that Kudu was still in the process of inserting
>>>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>>>
>>>>>
>>>>> Following up on what I posted, take a look at
>>>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>>>> read_operations_scans. It seems definitely possible that not all of
>>>>> the rows had finished inserting when counting, or that the scans were sent
>>>>> to a stale replica.
>>>>>
>>>>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong <aw...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Petter,
>>>>>>
>>>>>> When we verified that all data was inserted we found that some data
>>>>>>> was missing. We added this missing data and on some chunks we got the
>>>>>>> information that all rows were already present, i.e impala says something
>>>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>>>>> a different value.
>>>>>>
>>>>>>
>>>>>> How did you verify that all the data was inserted and how did you
>>>>>> find some data missing? I'm wondering if it's possible that the initial
>>>>>> "missing" data was data that Kudu was still in the process of inserting
>>>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>>>
>>>>>> Now to my question. Will data be inconsistent if we recycle Kudu
>>>>>>> after seeing soft memory limit warnings?
>>>>>>
>>>>>>
>>>>>> Your data should be consistently written, even with those warnings.
>>>>>> AFAIK they would cause a bit of slowness, not incorrect results.
>>>>>>
>>>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>>>>> restart the tablet servers, only restart one tablet server at a time or
>>>>>>> something like that)?
>>>>>>
>>>>>>
>>>>>> In general, you can use the `ksck` tool to check the health of your
>>>>>> cluster. See https://kudu.apache.org/docs/command_line_tools_referenc
>>>>>> e.html#cluster-ksck for more details. For restarting a cluster, I
>>>>>> would recommend taking down all tablet servers at once, otherwise tablet
>>>>>> replicas may try to replicate data from the server that was taken down.
>>>>>>
>>>>>> Hope this helped,
>>>>>> Andrew
>>>>>>
>>>>>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
>>>>>> petter.von.dolwitz@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Kudu users,
>>>>>>>
>>>>>>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline
>>>>>>> for evaluation we ingested 3 month worth of data. During ingestion we were
>>>>>>> facing messages from the maintenance threads that a soft memory limit were
>>>>>>> reached. It seems like the background maintenance threads stopped
>>>>>>> performing their tasks at this point in time. It also so seems like the
>>>>>>> memory was never recovered even after stopping ingestion so I guess there
>>>>>>> was a large backlog being built up. I guess the root cause here is that we
>>>>>>> were a bit too conservative when giving Kudu memory. After a reststart a
>>>>>>> lot of maintenance tasks were started (i.e. compaction).
>>>>>>>
>>>>>>> When we verified that all data was inserted we found that some data
>>>>>>> was missing. We added this missing data and on some chunks we got the
>>>>>>> information that all rows were already present, i.e impala says something
>>>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>>>>> a different value.
>>>>>>>
>>>>>>> Now to my question. Will data be inconsistent if we recycle Kudu
>>>>>>> after seeing soft memory limit warnings?
>>>>>>>
>>>>>>> Is there a way to tell when it is safe to restart Kudu to avoid
>>>>>>> these issues? Should we use any special procedure when restarting (e.g.
>>>>>>> only restart the tablet servers, only restart one tablet server at a time
>>>>>>> or something like that)?
>>>>>>>
>>>>>>> The table design uses 50 tablets per day (times 90 days). It is 8 TB
>>>>>>> of data after 3xreplication over 5 tablet servers.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Petter
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Andrew Wong
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Andrew Wong
>>>>>
>>>>
>>>>
>>>
>>
>
--
Andrew Wong
Re: Data inconsistency after restart
Posted by Boris Tyukin <bo...@boristyukin.com>.
this is smart, we are doing the same thing but the best part that attracts
me to Kudu is replacing our main HDFS storage with Kudu to enable near RT
use cases and not to deal with HBase and a Lambda architecture mess so
reliability and scalability is a big deal for us as we are looking to move
most of our data to Kudu.
On Wed, Dec 6, 2017 at 9:58 AM, Petter von Dolwitz (Hem) <
petter.von.dolwitz@gmail.com> wrote:
> Hi Boris,
>
> we do not have a Cloudera contract at the moment. Until we gained more
> Kudu experience we keep our master data in parquet format so that we can
> rebuild Kudu-tables upon errors. We are still in the early learning phase.
>
> Br,
> Petter
>
>
>
> 2017-12-06 14:35 GMT+01:00 Boris Tyukin <bo...@boristyukin.com>:
>
>> this is definitely concerning thread for us looking to use Impala for
>> storing mission-critical company data. Petter, are you paid Cloudera
>> customer btw? I wonder if you opened support ticket as well
>>
>> On Wed, Dec 6, 2017 at 7:26 AM, Petter von Dolwitz (Hem) <
>> petter.von.dolwitz@gmail.com> wrote:
>>
>>> Thanks for your reply Andrew!
>>>
>>> >How did you verify that all the data was inserted and how did you find
>>> some data missing?
>>> This was done using Impala. We counted the rows for groups representing
>>> the chunks we inserted.
>>>
>>> >Following up on what I posted, take a look at
>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>> read_operations_scans. It seems definitely possible that not all of the
>>> rows had finished inserting when counting, or that the scans were sent to a
>>> stale replica.
>>> Before we shut down we could only see the following in the logs. I.e.,
>>> no sign that ingestion was still ongoing.
>>>
>>> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
>>> 07:27:35.010694 90793 maintenance_manager.cc:383] P
>>> a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft memory
>>> limit (current capacity is 67.52%). However, there are no ops currently
>>> runnable which would free memory.
>>>
>>> Also the (cloudera) metric total_kudu_rows_inserted_rate_across_kudu_replicas
>>> showed zero.
>>>
>>> Still it seems like some data became inconsistent after restart. But if
>>> the maintenance_manager performs important jobs that are required to ensure
>>> that all data is inserted then I can understand why we ended up with
>>> inconsistent data. But, if I understand you correct, you are saying that
>>> these jobs are not critical for ingestion. In the link you provided I read
>>> "Impala scans are currently performed as READ_LATEST and have no
>>> consistency guarantees.". I would assume this means that it does not
>>> guarantee consistency if new data is inserted but should give valid (and
>>> same) results if no new data is inserted?
>>>
>>> I have not tried the ksck tool yet. Thank you for reminding. I will have
>>> a look.
>>>
>>> Br,
>>> Petter
>>>
>>>
>>> 2017-12-06 1:31 GMT+01:00 Andrew Wong <aw...@cloudera.com>:
>>>
>>>> How did you verify that all the data was inserted and how did you find
>>>>> some data missing? I'm wondering if it's possible that the initial
>>>>> "missing" data was data that Kudu was still in the process of inserting
>>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>>
>>>>
>>>> Following up on what I posted, take a look at
>>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>>> read_operations_scans. It seems definitely possible that not all of
>>>> the rows had finished inserting when counting, or that the scans were sent
>>>> to a stale replica.
>>>>
>>>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong <aw...@cloudera.com> wrote:
>>>>
>>>>> Hi Petter,
>>>>>
>>>>> When we verified that all data was inserted we found that some data
>>>>>> was missing. We added this missing data and on some chunks we got the
>>>>>> information that all rows were already present, i.e impala says something
>>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>>>> a different value.
>>>>>
>>>>>
>>>>> How did you verify that all the data was inserted and how did you find
>>>>> some data missing? I'm wondering if it's possible that the initial
>>>>> "missing" data was data that Kudu was still in the process of inserting
>>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>>
>>>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>>>>> seeing soft memory limit warnings?
>>>>>
>>>>>
>>>>> Your data should be consistently written, even with those warnings.
>>>>> AFAIK they would cause a bit of slowness, not incorrect results.
>>>>>
>>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>>>> restart the tablet servers, only restart one tablet server at a time or
>>>>>> something like that)?
>>>>>
>>>>>
>>>>> In general, you can use the `ksck` tool to check the health of your
>>>>> cluster. See https://kudu.apache.org/docs/command_line_tools_referenc
>>>>> e.html#cluster-ksck for more details. For restarting a cluster, I
>>>>> would recommend taking down all tablet servers at once, otherwise tablet
>>>>> replicas may try to replicate data from the server that was taken down.
>>>>>
>>>>> Hope this helped,
>>>>> Andrew
>>>>>
>>>>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
>>>>> petter.von.dolwitz@gmail.com> wrote:
>>>>>
>>>>>> Hi Kudu users,
>>>>>>
>>>>>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline for
>>>>>> evaluation we ingested 3 month worth of data. During ingestion we were
>>>>>> facing messages from the maintenance threads that a soft memory limit were
>>>>>> reached. It seems like the background maintenance threads stopped
>>>>>> performing their tasks at this point in time. It also so seems like the
>>>>>> memory was never recovered even after stopping ingestion so I guess there
>>>>>> was a large backlog being built up. I guess the root cause here is that we
>>>>>> were a bit too conservative when giving Kudu memory. After a reststart a
>>>>>> lot of maintenance tasks were started (i.e. compaction).
>>>>>>
>>>>>> When we verified that all data was inserted we found that some data
>>>>>> was missing. We added this missing data and on some chunks we got the
>>>>>> information that all rows were already present, i.e impala says something
>>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>>>> a different value.
>>>>>>
>>>>>> Now to my question. Will data be inconsistent if we recycle Kudu
>>>>>> after seeing soft memory limit warnings?
>>>>>>
>>>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>>>> restart the tablet servers, only restart one tablet server at a time or
>>>>>> something like that)?
>>>>>>
>>>>>> The table design uses 50 tablets per day (times 90 days). It is 8 TB
>>>>>> of data after 3xreplication over 5 tablet servers.
>>>>>>
>>>>>> Thanks,
>>>>>> Petter
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Andrew Wong
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Andrew Wong
>>>>
>>>
>>>
>>
>
Re: Data inconsistency after restart
Posted by "Petter von Dolwitz (Hem)" <pe...@gmail.com>.
Hi Boris,
we do not have a Cloudera contract at the moment. Until we gained more Kudu
experience we keep our master data in parquet format so that we can rebuild
Kudu-tables upon errors. We are still in the early learning phase.
Br,
Petter
2017-12-06 14:35 GMT+01:00 Boris Tyukin <bo...@boristyukin.com>:
> this is definitely concerning thread for us looking to use Impala for
> storing mission-critical company data. Petter, are you paid Cloudera
> customer btw? I wonder if you opened support ticket as well
>
> On Wed, Dec 6, 2017 at 7:26 AM, Petter von Dolwitz (Hem) <
> petter.von.dolwitz@gmail.com> wrote:
>
>> Thanks for your reply Andrew!
>>
>> >How did you verify that all the data was inserted and how did you find
>> some data missing?
>> This was done using Impala. We counted the rows for groups representing
>> the chunks we inserted.
>>
>> >Following up on what I posted, take a look at
>> https://kudu.apache.org/docs/transaction_semantics.html#_
>> read_operations_scans. It seems definitely possible that not all of the
>> rows had finished inserting when counting, or that the scans were sent to a
>> stale replica.
>> Before we shut down we could only see the following in the logs. I.e., no
>> sign that ingestion was still ongoing.
>>
>> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
>> 07:27:35.010694 90793 maintenance_manager.cc:383] P
>> a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft memory limit
>> (current capacity is 67.52%). However, there are no ops currently runnable
>> which would free memory.
>>
>> Also the (cloudera) metric total_kudu_rows_inserted_rate_across_kudu_replicas
>> showed zero.
>>
>> Still it seems like some data became inconsistent after restart. But if
>> the maintenance_manager performs important jobs that are required to ensure
>> that all data is inserted then I can understand why we ended up with
>> inconsistent data. But, if I understand you correct, you are saying that
>> these jobs are not critical for ingestion. In the link you provided I read
>> "Impala scans are currently performed as READ_LATEST and have no
>> consistency guarantees.". I would assume this means that it does not
>> guarantee consistency if new data is inserted but should give valid (and
>> same) results if no new data is inserted?
>>
>> I have not tried the ksck tool yet. Thank you for reminding. I will have
>> a look.
>>
>> Br,
>> Petter
>>
>>
>> 2017-12-06 1:31 GMT+01:00 Andrew Wong <aw...@cloudera.com>:
>>
>>> How did you verify that all the data was inserted and how did you find
>>>> some data missing? I'm wondering if it's possible that the initial
>>>> "missing" data was data that Kudu was still in the process of inserting
>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>
>>>
>>> Following up on what I posted, take a look at
>>> https://kudu.apache.org/docs/transaction_semantics.html#_
>>> read_operations_scans. It seems definitely possible that not all of the
>>> rows had finished inserting when counting, or that the scans were sent to a
>>> stale replica.
>>>
>>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong <aw...@cloudera.com> wrote:
>>>
>>>> Hi Petter,
>>>>
>>>> When we verified that all data was inserted we found that some data was
>>>>> missing. We added this missing data and on some chunks we got the
>>>>> information that all rows were already present, i.e impala says something
>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>>> a different value.
>>>>
>>>>
>>>> How did you verify that all the data was inserted and how did you find
>>>> some data missing? I'm wondering if it's possible that the initial
>>>> "missing" data was data that Kudu was still in the process of inserting
>>>> (albeit slowly, due to memory backpressure or somesuch).
>>>>
>>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>>>> seeing soft memory limit warnings?
>>>>
>>>>
>>>> Your data should be consistently written, even with those warnings.
>>>> AFAIK they would cause a bit of slowness, not incorrect results.
>>>>
>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>>> restart the tablet servers, only restart one tablet server at a time or
>>>>> something like that)?
>>>>
>>>>
>>>> In general, you can use the `ksck` tool to check the health of your
>>>> cluster. See https://kudu.apache.org/docs/command_line_tools_referenc
>>>> e.html#cluster-ksck for more details. For restarting a cluster, I
>>>> would recommend taking down all tablet servers at once, otherwise tablet
>>>> replicas may try to replicate data from the server that was taken down.
>>>>
>>>> Hope this helped,
>>>> Andrew
>>>>
>>>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
>>>> petter.von.dolwitz@gmail.com> wrote:
>>>>
>>>>> Hi Kudu users,
>>>>>
>>>>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline for
>>>>> evaluation we ingested 3 month worth of data. During ingestion we were
>>>>> facing messages from the maintenance threads that a soft memory limit were
>>>>> reached. It seems like the background maintenance threads stopped
>>>>> performing their tasks at this point in time. It also so seems like the
>>>>> memory was never recovered even after stopping ingestion so I guess there
>>>>> was a large backlog being built up. I guess the root cause here is that we
>>>>> were a bit too conservative when giving Kudu memory. After a reststart a
>>>>> lot of maintenance tasks were started (i.e. compaction).
>>>>>
>>>>> When we verified that all data was inserted we found that some data
>>>>> was missing. We added this missing data and on some chunks we got the
>>>>> information that all rows were already present, i.e impala says something
>>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>>> a different value.
>>>>>
>>>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>>>> seeing soft memory limit warnings?
>>>>>
>>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>>> restart the tablet servers, only restart one tablet server at a time or
>>>>> something like that)?
>>>>>
>>>>> The table design uses 50 tablets per day (times 90 days). It is 8 TB
>>>>> of data after 3xreplication over 5 tablet servers.
>>>>>
>>>>> Thanks,
>>>>> Petter
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Andrew Wong
>>>>
>>>
>>>
>>>
>>> --
>>> Andrew Wong
>>>
>>
>>
>
Re: Data inconsistency after restart
Posted by Boris Tyukin <bo...@boristyukin.com>.
this is definitely concerning thread for us looking to use Impala for
storing mission-critical company data. Petter, are you paid Cloudera
customer btw? I wonder if you opened support ticket as well
On Wed, Dec 6, 2017 at 7:26 AM, Petter von Dolwitz (Hem) <
petter.von.dolwitz@gmail.com> wrote:
> Thanks for your reply Andrew!
>
> >How did you verify that all the data was inserted and how did you find
> some data missing?
> This was done using Impala. We counted the rows for groups representing
> the chunks we inserted.
>
> >Following up on what I posted, take a look at https://kudu.apache.org/doc
> s/transaction_semantics.html#_read_operations_scans. It seems definitely
> possible that not all of the rows had finished inserting when counting, or
> that the scans were sent to a stale replica.
> Before we shut down we could only see the following in the logs. I.e., no
> sign that ingestion was still ongoing.
>
> kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
> 07:27:35.010694 90793 maintenance_manager.cc:383] P
> a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft memory limit
> (current capacity is 67.52%). However, there are no ops currently runnable
> which would free memory.
>
> Also the (cloudera) metric total_kudu_rows_inserted_rate_across_kudu_replicas
> showed zero.
>
> Still it seems like some data became inconsistent after restart. But if
> the maintenance_manager performs important jobs that are required to ensure
> that all data is inserted then I can understand why we ended up with
> inconsistent data. But, if I understand you correct, you are saying that
> these jobs are not critical for ingestion. In the link you provided I read
> "Impala scans are currently performed as READ_LATEST and have no
> consistency guarantees.". I would assume this means that it does not
> guarantee consistency if new data is inserted but should give valid (and
> same) results if no new data is inserted?
>
> I have not tried the ksck tool yet. Thank you for reminding. I will have a
> look.
>
> Br,
> Petter
>
>
> 2017-12-06 1:31 GMT+01:00 Andrew Wong <aw...@cloudera.com>:
>
>> How did you verify that all the data was inserted and how did you find
>>> some data missing? I'm wondering if it's possible that the initial
>>> "missing" data was data that Kudu was still in the process of inserting
>>> (albeit slowly, due to memory backpressure or somesuch).
>>>
>>
>> Following up on what I posted, take a look at https://kudu.apache.org/doc
>> s/transaction_semantics.html#_read_operations_scans. It seems definitely
>> possible that not all of the rows had finished inserting when counting, or
>> that the scans were sent to a stale replica.
>>
>> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong <aw...@cloudera.com> wrote:
>>
>>> Hi Petter,
>>>
>>> When we verified that all data was inserted we found that some data was
>>>> missing. We added this missing data and on some chunks we got the
>>>> information that all rows were already present, i.e impala says something
>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>> a different value.
>>>
>>>
>>> How did you verify that all the data was inserted and how did you find
>>> some data missing? I'm wondering if it's possible that the initial
>>> "missing" data was data that Kudu was still in the process of inserting
>>> (albeit slowly, due to memory backpressure or somesuch).
>>>
>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>>> seeing soft memory limit warnings?
>>>
>>>
>>> Your data should be consistently written, even with those warnings.
>>> AFAIK they would cause a bit of slowness, not incorrect results.
>>>
>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>> restart the tablet servers, only restart one tablet server at a time or
>>>> something like that)?
>>>
>>>
>>> In general, you can use the `ksck` tool to check the health of your
>>> cluster. See https://kudu.apache.org/docs/command_line_tools_referenc
>>> e.html#cluster-ksck for more details. For restarting a cluster, I would
>>> recommend taking down all tablet servers at once, otherwise tablet replicas
>>> may try to replicate data from the server that was taken down.
>>>
>>> Hope this helped,
>>> Andrew
>>>
>>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
>>> petter.von.dolwitz@gmail.com> wrote:
>>>
>>>> Hi Kudu users,
>>>>
>>>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline for
>>>> evaluation we ingested 3 month worth of data. During ingestion we were
>>>> facing messages from the maintenance threads that a soft memory limit were
>>>> reached. It seems like the background maintenance threads stopped
>>>> performing their tasks at this point in time. It also so seems like the
>>>> memory was never recovered even after stopping ingestion so I guess there
>>>> was a large backlog being built up. I guess the root cause here is that we
>>>> were a bit too conservative when giving Kudu memory. After a reststart a
>>>> lot of maintenance tasks were started (i.e. compaction).
>>>>
>>>> When we verified that all data was inserted we found that some data was
>>>> missing. We added this missing data and on some chunks we got the
>>>> information that all rows were already present, i.e impala says something
>>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>>> shows that the Kudu table is complete. So, even though we did not insert
>>>> any data on some chunks, a count(*) operation over these chunks now returns
>>>> a different value.
>>>>
>>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>>> seeing soft memory limit warnings?
>>>>
>>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>>> issues? Should we use any special procedure when restarting (e.g. only
>>>> restart the tablet servers, only restart one tablet server at a time or
>>>> something like that)?
>>>>
>>>> The table design uses 50 tablets per day (times 90 days). It is 8 TB of
>>>> data after 3xreplication over 5 tablet servers.
>>>>
>>>> Thanks,
>>>> Petter
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Andrew Wong
>>>
>>
>>
>>
>> --
>> Andrew Wong
>>
>
>
Re: Data inconsistency after restart
Posted by "Petter von Dolwitz (Hem)" <pe...@gmail.com>.
Thanks for your reply Andrew!
>How did you verify that all the data was inserted and how did you find
some data missing?
This was done using Impala. We counted the rows for groups representing the
chunks we inserted.
>Following up on what I posted, take a look at https://kudu.apache.org/
docs/transaction_semantics.html#_read_operations_scans. It seems definitely
possible that not all of the rows had finished inserting when counting, or
that the scans were sent to a stale replica.
Before we shut down we could only see the following in the logs. I.e., no
sign that ingestion was still ongoing.
kudu-tserver.ip-xx-yyy-z-nnn.root.log.INFO.20171201-065232.90314:I1201
07:27:35.010694 90793 maintenance_manager.cc:383] P
a38902afefca4a85a5469d149df9b4cb: we have exceeded our soft memory limit
(current capacity is 67.52%). However, there are no ops currently runnable
which would free memory.
Also the (cloudera) metric
total_kudu_rows_inserted_rate_across_kudu_replicas showed zero.
Still it seems like some data became inconsistent after restart. But if the
maintenance_manager performs important jobs that are required to ensure
that all data is inserted then I can understand why we ended up with
inconsistent data. But, if I understand you correct, you are saying that
these jobs are not critical for ingestion. In the link you provided I read
"Impala scans are currently performed as READ_LATEST and have no
consistency guarantees.". I would assume this means that it does not
guarantee consistency if new data is inserted but should give valid (and
same) results if no new data is inserted?
I have not tried the ksck tool yet. Thank you for reminding. I will have a
look.
Br,
Petter
2017-12-06 1:31 GMT+01:00 Andrew Wong <aw...@cloudera.com>:
> How did you verify that all the data was inserted and how did you find
>> some data missing? I'm wondering if it's possible that the initial
>> "missing" data was data that Kudu was still in the process of inserting
>> (albeit slowly, due to memory backpressure or somesuch).
>>
>
> Following up on what I posted, take a look at https://kudu.apache.org/
> docs/transaction_semantics.html#_read_operations_scans. It seems
> definitely possible that not all of the rows had finished inserting when
> counting, or that the scans were sent to a stale replica.
>
> On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong <aw...@cloudera.com> wrote:
>
>> Hi Petter,
>>
>> When we verified that all data was inserted we found that some data was
>>> missing. We added this missing data and on some chunks we got the
>>> information that all rows were already present, i.e impala says something
>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>> shows that the Kudu table is complete. So, even though we did not insert
>>> any data on some chunks, a count(*) operation over these chunks now returns
>>> a different value.
>>
>>
>> How did you verify that all the data was inserted and how did you find
>> some data missing? I'm wondering if it's possible that the initial
>> "missing" data was data that Kudu was still in the process of inserting
>> (albeit slowly, due to memory backpressure or somesuch).
>>
>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>> seeing soft memory limit warnings?
>>
>>
>> Your data should be consistently written, even with those warnings. AFAIK
>> they would cause a bit of slowness, not incorrect results.
>>
>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>> issues? Should we use any special procedure when restarting (e.g. only
>>> restart the tablet servers, only restart one tablet server at a time or
>>> something like that)?
>>
>>
>> In general, you can use the `ksck` tool to check the health of your
>> cluster. See https://kudu.apache.org/docs/command_line_tools_referenc
>> e.html#cluster-ksck for more details. For restarting a cluster, I would
>> recommend taking down all tablet servers at once, otherwise tablet replicas
>> may try to replicate data from the server that was taken down.
>>
>> Hope this helped,
>> Andrew
>>
>> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
>> petter.von.dolwitz@gmail.com> wrote:
>>
>>> Hi Kudu users,
>>>
>>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline for
>>> evaluation we ingested 3 month worth of data. During ingestion we were
>>> facing messages from the maintenance threads that a soft memory limit were
>>> reached. It seems like the background maintenance threads stopped
>>> performing their tasks at this point in time. It also so seems like the
>>> memory was never recovered even after stopping ingestion so I guess there
>>> was a large backlog being built up. I guess the root cause here is that we
>>> were a bit too conservative when giving Kudu memory. After a reststart a
>>> lot of maintenance tasks were started (i.e. compaction).
>>>
>>> When we verified that all data was inserted we found that some data was
>>> missing. We added this missing data and on some chunks we got the
>>> information that all rows were already present, i.e impala says something
>>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>>> shows that the Kudu table is complete. So, even though we did not insert
>>> any data on some chunks, a count(*) operation over these chunks now returns
>>> a different value.
>>>
>>> Now to my question. Will data be inconsistent if we recycle Kudu after
>>> seeing soft memory limit warnings?
>>>
>>> Is there a way to tell when it is safe to restart Kudu to avoid these
>>> issues? Should we use any special procedure when restarting (e.g. only
>>> restart the tablet servers, only restart one tablet server at a time or
>>> something like that)?
>>>
>>> The table design uses 50 tablets per day (times 90 days). It is 8 TB of
>>> data after 3xreplication over 5 tablet servers.
>>>
>>> Thanks,
>>> Petter
>>>
>>>
>>>
>>
>>
>> --
>> Andrew Wong
>>
>
>
>
> --
> Andrew Wong
>
Re: Data inconsistency after restart
Posted by Andrew Wong <aw...@cloudera.com>.
>
> How did you verify that all the data was inserted and how did you find
> some data missing? I'm wondering if it's possible that the initial
> "missing" data was data that Kudu was still in the process of inserting
> (albeit slowly, due to memory backpressure or somesuch).
>
Following up on what I posted, take a look at
https://kudu.apache.org/docs/transaction_semantics.html#_read_operations_scans.
It seems definitely possible that not all of the rows had finished
inserting when counting, or that the scans were sent to a stale replica.
On Tue, Dec 5, 2017 at 4:18 PM, Andrew Wong <aw...@cloudera.com> wrote:
> Hi Petter,
>
> When we verified that all data was inserted we found that some data was
>> missing. We added this missing data and on some chunks we got the
>> information that all rows were already present, i.e impala says something
>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>> shows that the Kudu table is complete. So, even though we did not insert
>> any data on some chunks, a count(*) operation over these chunks now returns
>> a different value.
>
>
> How did you verify that all the data was inserted and how did you find
> some data missing? I'm wondering if it's possible that the initial
> "missing" data was data that Kudu was still in the process of inserting
> (albeit slowly, due to memory backpressure or somesuch).
>
> Now to my question. Will data be inconsistent if we recycle Kudu after
>> seeing soft memory limit warnings?
>
>
> Your data should be consistently written, even with those warnings. AFAIK
> they would cause a bit of slowness, not incorrect results.
>
> Is there a way to tell when it is safe to restart Kudu to avoid these
>> issues? Should we use any special procedure when restarting (e.g. only
>> restart the tablet servers, only restart one tablet server at a time or
>> something like that)?
>
>
> In general, you can use the `ksck` tool to check the health of your
> cluster. See https://kudu.apache.org/docs/command_line_tools_
> reference.html#cluster-ksck for more details. For restarting a cluster, I
> would recommend taking down all tablet servers at once, otherwise tablet
> replicas may try to replicate data from the server that was taken down.
>
> Hope this helped,
> Andrew
>
> On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
> petter.von.dolwitz@gmail.com> wrote:
>
>> Hi Kudu users,
>>
>> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline for
>> evaluation we ingested 3 month worth of data. During ingestion we were
>> facing messages from the maintenance threads that a soft memory limit were
>> reached. It seems like the background maintenance threads stopped
>> performing their tasks at this point in time. It also so seems like the
>> memory was never recovered even after stopping ingestion so I guess there
>> was a large backlog being built up. I guess the root cause here is that we
>> were a bit too conservative when giving Kudu memory. After a reststart a
>> lot of maintenance tasks were started (i.e. compaction).
>>
>> When we verified that all data was inserted we found that some data was
>> missing. We added this missing data and on some chunks we got the
>> information that all rows were already present, i.e impala says something
>> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
>> shows that the Kudu table is complete. So, even though we did not insert
>> any data on some chunks, a count(*) operation over these chunks now returns
>> a different value.
>>
>> Now to my question. Will data be inconsistent if we recycle Kudu after
>> seeing soft memory limit warnings?
>>
>> Is there a way to tell when it is safe to restart Kudu to avoid these
>> issues? Should we use any special procedure when restarting (e.g. only
>> restart the tablet servers, only restart one tablet server at a time or
>> something like that)?
>>
>> The table design uses 50 tablets per day (times 90 days). It is 8 TB of
>> data after 3xreplication over 5 tablet servers.
>>
>> Thanks,
>> Petter
>>
>>
>>
>
>
> --
> Andrew Wong
>
--
Andrew Wong
Re: Data inconsistency after restart
Posted by Andrew Wong <aw...@cloudera.com>.
Hi Petter,
When we verified that all data was inserted we found that some data was
> missing. We added this missing data and on some chunks we got the
> information that all rows were already present, i.e impala says something
> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
> shows that the Kudu table is complete. So, even though we did not insert
> any data on some chunks, a count(*) operation over these chunks now returns
> a different value.
How did you verify that all the data was inserted and how did you find some
data missing? I'm wondering if it's possible that the initial "missing"
data was data that Kudu was still in the process of inserting (albeit
slowly, due to memory backpressure or somesuch).
Now to my question. Will data be inconsistent if we recycle Kudu after
> seeing soft memory limit warnings?
Your data should be consistently written, even with those warnings. AFAIK
they would cause a bit of slowness, not incorrect results.
Is there a way to tell when it is safe to restart Kudu to avoid these
> issues? Should we use any special procedure when restarting (e.g. only
> restart the tablet servers, only restart one tablet server at a time or
> something like that)?
In general, you can use the `ksck` tool to check the health of your
cluster. See
https://kudu.apache.org/docs/command_line_tools_reference.html#cluster-ksck
for more details. For restarting a cluster, I would recommend taking down
all tablet servers at once, otherwise tablet replicas may try to replicate
data from the server that was taken down.
Hope this helped,
Andrew
On Tue, Dec 5, 2017 at 10:42 AM, Petter von Dolwitz (Hem) <
petter.von.dolwitz@gmail.com> wrote:
> Hi Kudu users,
>
> We just started to use Kudu (1.4.0+cdh5.12.1). To make a baseline for
> evaluation we ingested 3 month worth of data. During ingestion we were
> facing messages from the maintenance threads that a soft memory limit were
> reached. It seems like the background maintenance threads stopped
> performing their tasks at this point in time. It also so seems like the
> memory was never recovered even after stopping ingestion so I guess there
> was a large backlog being built up. I guess the root cause here is that we
> were a bit too conservative when giving Kudu memory. After a reststart a
> lot of maintenance tasks were started (i.e. compaction).
>
> When we verified that all data was inserted we found that some data was
> missing. We added this missing data and on some chunks we got the
> information that all rows were already present, i.e impala says something
> like Modified: 0 rows, nnnnnnn errors. Doing the verification again now
> shows that the Kudu table is complete. So, even though we did not insert
> any data on some chunks, a count(*) operation over these chunks now returns
> a different value.
>
> Now to my question. Will data be inconsistent if we recycle Kudu after
> seeing soft memory limit warnings?
>
> Is there a way to tell when it is safe to restart Kudu to avoid these
> issues? Should we use any special procedure when restarting (e.g. only
> restart the tablet servers, only restart one tablet server at a time or
> something like that)?
>
> The table design uses 50 tablets per day (times 90 days). It is 8 TB of
> data after 3xreplication over 5 tablet servers.
>
> Thanks,
> Petter
>
>
>
--
Andrew Wong