You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/04/02 10:11:41 UTC
[GitHub] [superset] slim-patchy opened a new issue #13930: Add support of MongoDB to Superset via a new connector
slim-patchy opened a new issue #13930:
URL: https://github.com/apache/superset/issues/13930
I am interested in implementing a new connector for Superset to add MongoDB support, and would like to know if others have started similar work too so as to avoid duplicate effort.
### Inspiration
@priyankajuyal I think there're 3 methods to support mongodb in superset:
- Implement a mongodb in sqlalchemy, this requires a lot of work and may not be easy to do cause sqlalchemy is a object relational mapper library, while mongodb is not relational.
- Implement a new connector for superset. Right now superset has two connectors, one is for sqlalchemy, one is for druid, and similar things can be implemented for mongodb, this should be the right choice.
- Transform mongodb to some relational db, stripe has a deprecated project for this job: https://github.com/stripe/mosql.
_Originally posted by @xiaohanyu in https://github.com/apache/superset/issues/4231#issuecomment-358608747_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] slim-patchy commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
slim-patchy commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-826487947
I have just found out about PartiQL today: https://partiql.org/
It is a querying language for relational, semi-structured, and nested data. So it supports NoSQL databases as well. I will see if I could write a PartiQL connector for Superset to fulfil the requirements of this issue. But it does look like a viable candidate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] slim-patchy commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
slim-patchy commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-830778681
Sure @srinify
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] jbouzekri commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
jbouzekri commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-892127482
It seems to fallback to a BaseEngineSpec instance : https://github.com/apache/superset/blob/master/superset/db_engine_specs/base.py#L148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] slim-patchy commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
slim-patchy commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-818718710
Yeah @srinify I can totally see where you are coming from. I think what I am proposing may be an entirely different variant of Superset if the Superset as we know it only relies on its SQLAlchemy connector, which enables support for SQL / Relational DBs in the first place, so it might require major work.
But instead my rationale for a NoSQL connector feeding data to the SQLAlchemy connector is that ultimately for data to be presented in a coherent way we really need highly structured data to be fed so that they can be visually represented in a very specific set of ways (say like as a dashboard). So I do not suppose I would bypass the existing SQLAlchemy connector, but rather add an adapter to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] slim-patchy commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
slim-patchy commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-840988048
Turned out PartiQL doesn't have a Python implementation yet, so am working on that at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] slim-patchy commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
slim-patchy commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-891362066
@jbouzekri Hey the work looks cool! Maybe I can also test it out for you soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] slim-patchy commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
slim-patchy commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-815967617
I have seen people attempted to come up with something based on the same idea with vastly different approaches. This appear to be something to be discussed / thought through really thoroughly. Stripe's archived project called `MoSQL` is one such more promising examples of others' previous attempts: https://github.com/stripe-archive/mosql, though that is written in Ruby. I think I could probably borrow some ideas for the PR concept from them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] slim-patchy commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
slim-patchy commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-826487947
I have just found out about PartiQL today: https://partiql.org/
It is a querying language for relational, semi-structured, and nested data. So it supports NoSQL databases as well. I will see if I could write a PartiQL connector for Superset to fulfil the requirements of this issue. But it does look like a viable candidate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] srinify commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
srinify commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-818189138
@slim-patchy to be honest I think your best bet is to use a query engine like Trino / PrestoDB / Drill. MongoDB AFAIK doesn't have a SQLAlchemy connector and it goes against all of the relational db principles that SQLAlchemy, Superset, etc are all built on & focused on supporting.
Originally, Druid didn't have SQL and a native connector was built for it. Since then, Druid learned to "speak SQL" (as Max says!) and the native connector got deprecated in favor of betting on the SQLAlchemy platform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] srinify commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
srinify commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-829358765
Keep us posted @slim-patchy !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] mistercrunch commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
mistercrunch commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-891556950
Oh very cool! How does this work without a specific `db_engine_spec` !? For reference, those are modules telling Superset about some specificities of the database engine, more about it here: https://github.com/apache/superset/tree/master/superset/db_engine_specs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] jbouzekri commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
jbouzekri commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-891319859
I just made a proof of concept of querying a mongo database with superset through [mongo bi connector](https://docs.mongodb.com/bi-connector/v2.14/) : https://github.com/jbouzekri/docker-superset-mongo-poc/. It seems to work pretty well. It is not in the documentation but I have tested row level security and it works too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] slim-patchy commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
slim-patchy commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-815962606
Thanks @srinify for your response!
I am not sure if the official MongoDB Python driver called `pymongo` is PEP 249 compliant. But I have been thinking that to make a new Superset connector work for MongoDB will have a slight different set of requirements, or even some schema-based mechanisms to bridge MongoDB with RDBs (instead). I have seen some academic papers discussing the nuances regarding this latter point. So I will need to really work out the concept behind such a connector first before I do any serious coding though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] srinify commented on issue #13930: Add support of MongoDB to Superset via a new connector
Posted by GitBox <gi...@apache.org>.
srinify commented on issue #13930:
URL: https://github.com/apache/superset/issues/13930#issuecomment-815871747
Hey @slim-patchy right now adding new connectors to Superset requires a SQLAlchemy + Python DB-API 2.0 compatible driver. Does Mongo have one?
https://preset.io/blog/building-database-connector/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org