Kudu/Spark LIMIT support

[Pavel Martynov <mr...@gmail.com>]

Hi, folks!

For testing purposes, I need to read a small chunk of rows of a big table
(~12 blns rows) on my dev machine. So I started driver with "local[4]"
executors and wrote a code like:

sparkSession.sqlContext.read.options(Map(
  "kudu.master" -> "master",
  "kudu.table" -> "thebigtable",
  "kudu.splitSizeBytes" -> SplitSize512Mb
)).format("kudu").load
  .limit(1000)
  .select($"col1", $"col2", $"col3")

My expectation: only 1000 rows should be actually read from Kudu in very
fast manner.

Actually observed: Spark started 4 parallel scanners for one of the tablets
and looks like this scanning process scanning the whole tablet (which is
~2.4 blns rows) and scanning time is really big.

Is this expected behavior?

I found this closed ticket https://issues.apache.org/jira/browse/KUDU-16 with
comments on Spark: "No support on the Spark side, but AFAICT, support for
limits given our current Scala bindings is somewhat unnatural.".

Kudu ver 1.11.1.

-- 
with best regards, Pavel Martynov

Re: Kudu/Spark LIMIT support

[Alexey Serbin <as...@cloudera.com>]

Yes, your observations match what's in the code: Kudu Spark bindings don't
support scanner row limits, but Kudu Java, C++ and Python clients do
support that.  And indeed,
https://issues.apache.org/jira/browse/KUDU-16 contains
relevant information on the status of this feature, missing

As of my knowledge, nobody currently works on implementing scanner
limits for kudu-spark.  However, patches are always welcome!


Kind regards,

Alexey

On Mon, Jan 20, 2020 at 10:38 PM Pavel Martynov <mr...@gmail.com> wrote:

> Hi, folks!
>
> For testing purposes, I need to read a small chunk of rows of a big table
> (~12 blns rows) on my dev machine. So I started driver with "local[4]"
> executors and wrote a code like:
>
> sparkSession.sqlContext.read.options(Map(
>   "kudu.master" -> "master",
>   "kudu.table" -> "thebigtable",
>   "kudu.splitSizeBytes" -> SplitSize512Mb
> )).format("kudu").load
>   .limit(1000)
>   .select($"col1", $"col2", $"col3")
>
> My expectation: only 1000 rows should be actually read from Kudu in very
> fast manner.
>
> Actually observed: Spark started 4 parallel scanners for one of the
> tablets and looks like this scanning process scanning the whole tablet
> (which is ~2.4 blns rows) and scanning time is really big.
>
> Is this expected behavior?
>
> I found this closed ticket https://issues.apache.org/jira/browse/KUDU-16 with
> comments on Spark: "No support on the Spark side, but AFAICT, support for
> limits given our current Scala bindings is somewhat unnatural.".
>
> Kudu ver 1.11.1.
>
> --
> with best regards, Pavel Martynov
>