KSQL to search for data in Kafka Topics

["M. Manna" <ma...@gmail.com>]

Hello,

I am quite new to KSQL, so apologise for misunderstanding it's concept.

I have a list of topics that I want to search data for. I am not using
stream process, but plain topics which has data retained for 14 days. All i
want to do is search for data in SQL-like way as long as it's within the 14
day window. Could I do that using KSQL, even when I am not using stream
topology?

Regards,

Re: KSQL to search for data in Kafka Topics

[Christopher Smith <cb...@gmail.com>]

I'm not sure what you mean by "not using stream topology". What does that
mean to you that you'd rather avoid?

However, you can indeed use KSQL to define streams & tables that process
data from a number of topics. However, I think you may have the
misimpression that KSQL is designed so you can execute queries like,
"select data from across all topics which meets this criteria". It's not.
Think of it as more like SQL, and each of your topics is effectively it's
own table. There's not a natural way to query all tables in SQL with a
single simple query, but it is possible to build queries that span any
number of tables. The equivalent is true in KSQL.

There may be a design issue here that you're struggling with, where data
that perhaps should really be in one topic is spread across a lot of
different topics. You can certainly connect/combine those topics in KSQL
(it's really quite good at that) and then apply whatever filtering/logic
you want from that.

--Chris

On Tue, May 19, 2020 at 7:35 AM M. Manna <ma...@gmail.com> wrote:

> Hello,
>
> I am quite new to KSQL, so apologise for misunderstanding it's concept.
>
> I have a list of topics that I want to search data for. I am not using
> stream process, but plain topics which has data retained for 14 days. All i
> want to do is search for data in SQL-like way as long as it's within the 14
> day window. Could I do that using KSQL, even when I am not using stream
> topology?
>
> Regards,
>


-- 
Chris