You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by hangu choi <de...@gmail.com> on 2015/10/20 08:47:16 UTC
Validating idea of architecture for RDB / Import / Solr
Hi,
I am newbie for solr and I hope to check my idea is good or terrible if
someone can help.
# background
* I have mysql as my primary data storage.
* I want to import data from mysql to solr (solrcloud).
* I have domain logics to make solr document - (means I can't make solr
document with plain sql query)
# my idea
* In mysql I make table for solr document, and insert/update whenever
any changes made for solr document.
* run data import handler (mostly delta query) from [solr document
table] to solr with cron scheduler frequently with soft-commit.
* hard-commit less frequently.
Is it good or terrible idea?
for any advice, I will be happy.
Regards,
Hangu
Re: Validating idea of architecture for RDB / Import / Solr
Posted by Hangu Choi <de...@gmail.com>.
Thank you Susheel, chandan.
Susheel,
In Option 1, I have to manage my domain logic two places, one for java, one
for sql.
and my domain logic is not stable and may change frequently.. I think
changing complext sql query whenever my domain logic change will be
painful...
and yes, I've thought like option 2. still my importing java application
has to change whenever domain logic change.
but because it is java, I can consider import domain logic or can make
domain layer as service, etc...
chandan, Thank you for sharing your experience.
Thank you all for your attention. It's helpful to me! :)
Regards,
Hangu
On Wed, Oct 21, 2015 at 1:01 AM, chandan khatri <ch...@gmail.com>
wrote:
> Hi Hongu,
>
> Scaling shouldn't be a problem if you follow the proposed approach but if
> the updates are frequent then there can be issue with high latency.
>
> I am also following the same approach where the data is first
> updated/written to Mysql and then in MongoDB. I believe using this we can
> take advantage of both RDBMS and NoSQL.
>
> I am only using MongoDB for non transactional data but planning to use it
> for transactional data e.g inventory.
>
> Thanks,
> Chandan
>
>
>
> On Tue, Oct 20, 2015 at 6:22 PM, Susheel Kumar <su...@gmail.com>
> wrote:
>
> > Hello Hangu,
> >
> >
> > OPTION1. you can write complex/nested join queries with DIH and have
> > functions written in javascript for transformations in data-config if
> that
> > meets your domain requirement
> > OPTION2. use Java program with SolrJ and read data using jdbc and apply
> > domain specific rules, create solr document and then using SolrJ classes
> > like (ClouldSolrClient...) write the documents to Solr. This can scale
> and
> > will have more flexibility as you go further.
> >
> > The current solution you are suggesting to go, can work depending on
> how
> > much data you are talking about / how many updates etc. but will have its
> > own consequences like maintaining a separate state of data (if it is
> large
> > you are storing it twice etc...), may not scale etc.
> >
> > Thanks,
> > Susheel
> >
> >
> >
> > On Tue, Oct 20, 2015 at 2:47 AM, hangu choi <de...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I am newbie for solr and I hope to check my idea is good or terrible if
> > > someone can help.
> > >
> > > # background
> > > * I have mysql as my primary data storage.
> > > * I want to import data from mysql to solr (solrcloud).
> > > * I have domain logics to make solr document - (means I can't make
> > solr
> > > document with plain sql query)
> > >
> > > # my idea
> > > * In mysql I make table for solr document, and insert/update
> whenever
> > > any changes made for solr document.
> > > * run data import handler (mostly delta query) from [solr document
> > > table] to solr with cron scheduler frequently with soft-commit.
> > > * hard-commit less frequently.
> > >
> > > Is it good or terrible idea?
> > > for any advice, I will be happy.
> > >
> > >
> > > Regards,
> > > Hangu
> > >
> >
>
Re: Validating idea of architecture for RDB / Import / Solr
Posted by chandan khatri <ch...@gmail.com>.
Hi Hongu,
Scaling shouldn't be a problem if you follow the proposed approach but if
the updates are frequent then there can be issue with high latency.
I am also following the same approach where the data is first
updated/written to Mysql and then in MongoDB. I believe using this we can
take advantage of both RDBMS and NoSQL.
I am only using MongoDB for non transactional data but planning to use it
for transactional data e.g inventory.
Thanks,
Chandan
On Tue, Oct 20, 2015 at 6:22 PM, Susheel Kumar <su...@gmail.com>
wrote:
> Hello Hangu,
>
>
> OPTION1. you can write complex/nested join queries with DIH and have
> functions written in javascript for transformations in data-config if that
> meets your domain requirement
> OPTION2. use Java program with SolrJ and read data using jdbc and apply
> domain specific rules, create solr document and then using SolrJ classes
> like (ClouldSolrClient...) write the documents to Solr. This can scale and
> will have more flexibility as you go further.
>
> The current solution you are suggesting to go, can work depending on how
> much data you are talking about / how many updates etc. but will have its
> own consequences like maintaining a separate state of data (if it is large
> you are storing it twice etc...), may not scale etc.
>
> Thanks,
> Susheel
>
>
>
> On Tue, Oct 20, 2015 at 2:47 AM, hangu choi <de...@gmail.com> wrote:
>
> > Hi,
> >
> > I am newbie for solr and I hope to check my idea is good or terrible if
> > someone can help.
> >
> > # background
> > * I have mysql as my primary data storage.
> > * I want to import data from mysql to solr (solrcloud).
> > * I have domain logics to make solr document - (means I can't make
> solr
> > document with plain sql query)
> >
> > # my idea
> > * In mysql I make table for solr document, and insert/update whenever
> > any changes made for solr document.
> > * run data import handler (mostly delta query) from [solr document
> > table] to solr with cron scheduler frequently with soft-commit.
> > * hard-commit less frequently.
> >
> > Is it good or terrible idea?
> > for any advice, I will be happy.
> >
> >
> > Regards,
> > Hangu
> >
>
Re: Validating idea of architecture for RDB / Import / Solr
Posted by Susheel Kumar <su...@gmail.com>.
Hello Hangu,
OPTION1. you can write complex/nested join queries with DIH and have
functions written in javascript for transformations in data-config if that
meets your domain requirement
OPTION2. use Java program with SolrJ and read data using jdbc and apply
domain specific rules, create solr document and then using SolrJ classes
like (ClouldSolrClient...) write the documents to Solr. This can scale and
will have more flexibility as you go further.
The current solution you are suggesting to go, can work depending on how
much data you are talking about / how many updates etc. but will have its
own consequences like maintaining a separate state of data (if it is large
you are storing it twice etc...), may not scale etc.
Thanks,
Susheel
On Tue, Oct 20, 2015 at 2:47 AM, hangu choi <de...@gmail.com> wrote:
> Hi,
>
> I am newbie for solr and I hope to check my idea is good or terrible if
> someone can help.
>
> # background
> * I have mysql as my primary data storage.
> * I want to import data from mysql to solr (solrcloud).
> * I have domain logics to make solr document - (means I can't make solr
> document with plain sql query)
>
> # my idea
> * In mysql I make table for solr document, and insert/update whenever
> any changes made for solr document.
> * run data import handler (mostly delta query) from [solr document
> table] to solr with cron scheduler frequently with soft-commit.
> * hard-commit less frequently.
>
> Is it good or terrible idea?
> for any advice, I will be happy.
>
>
> Regards,
> Hangu
>