[jira] [Commented] (KUDU-3747) Clarify on the C++ client meta-cache logic for create-table-and-immediate-write-and-scan use case

["Abhishek Chennaka (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125323#comment-18125323 ] 

Abhishek Chennaka commented on KUDU-3747:
-----------------------------------------

{quote}the client still calls KuduClient::Data::GetTabletServer() as a part of KuduScanner::Data::OpenTablet() when commencing read/scan operation.
{quote}
This call is normal. It runs once for every tablet on every scan, and it sends no request to the master. GetTabletServer() does two things which are both local:
  - It picks a replica from the tablet entry already in the meta-cache (SelectTServer).
  - It calls InitProxy(), which returns immediately when a connection object already exists (If there were previous writes onto this tablet).
I reproduced the repro.cc workload in a mini-cluster test (3 tservers, --raft_heartbeat_interval_ms=5000, 1 and 10 range partitions, LEADER_ONLY and CLOSEST_REPLICA). In every case the client sent zero GetTableLocations/GetTabletLocations RPCs to the master during Open() and NextBatch(), so the meta-cache served all lookups.

With a single partition there is no scan latency. With 10 partitions, NextBatch() takes ~4–6s. The number of partitions is not the cause but whether a tablet was written to. A newly elected leader doesn't contact its followers until a write arrives or its heartbeat timer fires (up to --raft_heartbeat_interval_ms), and until its first op commits its MVCC clean time is uninitialized
In case of the original 10 tablet situation the time goes into the sleep-and-retry loop at client side scanner [here|https://github.com/apache/kudu/blob/bdfb2b10cf5c5ce9eb67d0f669a5309d16b57ff4/src/kudu/client/scanner-internal.cc#L604-L616]:
  a. The client sends the scan to the leader (followers reject the same way with CLOSEST_REPLICA).
  b. The leader rejects it with clean time has not yet been initialized (TABLET_NOT_RUNNING).
  c. The client marks that leader as unusable (blacklists it).
  d. GetTabletServer() then finds no usable leader and returns No LEADER for tablet … (blacklist replicas …).
  e. The client sleeps and retries.
{quote}it would be natural for the client at least to rely on the information in the scan tokens when opening a table for scanning
{quote}
Even with tokens, the client would send the scan to the same leader and get the same rejection as tokens carry the locations the cache already has. This can be added as an enhancement changelist but doesn't change the behavior in repro.cc.

So I'd suggest doing the below in this Jira:
In scope of this Jira, we can add the following enhancement - Avoid roundtrips to master if leader replica is not needed by relying on meta-cache and let the scan-tokens prime the cache when cache is leaderless and the scan-token has a leader.

Create a follow-up Jira to fix the actual issue:
The safetime has to be advanced independent of the --raft_heartbeat_interval period.
Have the client report the underlying TABLET_NOT_RUNNING / clean-time error instead of "No LEADER for tablet …".

> Clarify on the C++ client meta-cache logic for create-table-and-immediate-write-and-scan use case
> -------------------------------------------------------------------------------------------------
>
>                 Key: KUDU-3747
>                 URL: https://issues.apache.org/jira/browse/KUDU-3747
>             Project: Kudu
>          Issue Type: Task
>          Components: client
>    Affects Versions: 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.17.1, 1.18.1
>            Reporter: Alexey Serbin
>            Priority: Major
>         Attachments: repro.cc
>
>
> With KUDU-1802 implemented, the Kudu C++ client is able to obtain information on tablet replica locations from scan tokens, avoiding extra calls to Kudu masters when such information in present.  It works very well in the majority of use cases.  However, a couple of surprises were discovered in a relatively rare use case when a Kudu client application creates a new table, immediately start writing data into the newly created table, and then reads data back from the table.
> The two unexpected items are:
> # After populating its metacache with the information on the location of tablet replicas and using it while writing data into the newly created table, the client still calls {{KuduClient::Data::GetTabletServer()}} as a part of {{KuduScanner::Data::OpenTablet()}} when commencing read/scan operation.
> # Even if ignoring the already existing tablet location information in the metacache (say, if it's short-lived and already expired), it would be natural for the client at least to rely on the information in the scan tokens when opening a table for scanning (given KUDU-1802 provisions are already there) instead of re-resolving tablet locations.
> This became apparent once increasing the setting for the {{\-\-raft_heartbeat_interval_ms}} setting that Kudu tablet servers run tenfold (i.e. from default 500 ms to 5000 ms): the client application that tries to exercise the described use case exhibited high latency when calling {{KuduScanner::Open()}} right after writing a few rows into the newly created table.
> It's necessary to clarify what's going on.
> [Attached|https://issues.apache.org/jira/secure/attachment/13081007/repro.cc] is a sample Kudu client application in C++.  Compile it similarly to {{$KUDU_HOME/examples/cpp/example.cc}} and run against a Kudu cluster having at least three tablet servers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)