Huge data and Superset Apache

[Mohammed Alqaisi <mo...@latro.com>]

Dear Sir/Madam

I hope this email finds you well,
I have been exploring Superset Apache for a while now, I have already connected our PostgreSQL database using the DB driver and I was wondering if the tool is able to create and visualize a dashboard to our database that contains almost 1 billion records? Is this something the tool can process and handle? It is important to us the speed of handling all these records and to filter out anything the client wants, and the main problem I'm facing is whenever I'm trying to create a chart it terminates and displays errors 1001 and 1002, Is there something we can do to resolve this issue?

I hope to hear from you,

Regards,
Mohammed

Re: Huge data and Superset Apache

[Ville Brofeldt <vi...@gmail.com>]

Hi Mohammed,

Superset defers “heavy lifting” to the analytical database, and in that sense there really isn’t an upper limit on the amount of data that Superset queries. Let’s assume you have 1 billion rows of IOT data from three different sensors, and want to visualise the number of records per sensor id with a pie chart. In this case you would be querying 1 billion rows of data, but only get back *three* rows from the database - one row per sensor. Therefore, from the perspective of Superset, it doesn’t really matter if your underlying table has 1 million or 100 billion rows - just as long as your analytical database can process the amount of data in an acceptable timeframe.

Ville

> On 15. Mar 2022, at 10.36, Mohammed Alqaisi <mo...@latro.com> wrote:
> 
> Dear Sir/Madam
> 
> I hope this email finds you well,
> I have been exploring Superset Apache for a while now, I have already connected our PostgreSQL database using the DB driver and I was wondering if the tool is able to create and visualize a dashboard to our database that contains almost 1 billion records? Is this something the tool can process and handle? It is important to us the speed of handling all these records and to filter out anything the client wants, and the main problem I'm facing is whenever I'm trying to create a chart it terminates and displays errors 1001 and 1002, Is there something we can do to resolve this issue?
> 
> I hope to hear from you,
> 
> Regards,
> Mohammed