You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by GitBox <gi...@apache.org> on 2021/05/27 00:52:04 UTC
[GitHub] [lucenenet] ensemblebd opened a new issue #489: question - large size recommendation?
ensemblebd opened a new issue #489:
URL: https://github.com/apache/lucenenet/issues/489
Hello.
I'm planning to use this for a massive amount of documents (millions).
And currently using RAMDirectory with a background FSDirectory sync operation on an interval to persist.
Would you recommend splitting documents by <T> into separate directories to keep the system fast, or would 10 x 100,000 per <T> not be an issue rolled into a single FSD?
Trying to plan ahead on theory. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] eladmarg commented on issue #489: question - large size recommendation?
Posted by GitBox <gi...@apache.org>.
eladmarg commented on issue #489:
URL: https://github.com/apache/lucenenet/issues/489#issuecomment-849317980
this strategy called sharding, it's better to split the data by your logic and how the users query it.
for example, if you have geolocation users and they using data most of the time for one country,
so you create an index for all new york and another one for new jersey.
if a user queries more than one location, you run both of the queries in parallel and aggregate the result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] ensemblebd commented on issue #489: question - large size recommendation?
Posted by GitBox <gi...@apache.org>.
ensemblebd commented on issue #489:
URL: https://github.com/apache/lucenenet/issues/489#issuecomment-849320902
Thank you! I hadn't thought of it in that perspective.
Will need to rethink what chunks I could fashion out of which data parts appropriately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [lucenenet] ensemblebd closed issue #489: question - large size recommendation?
Posted by GitBox <gi...@apache.org>.
ensemblebd closed issue #489:
URL: https://github.com/apache/lucenenet/issues/489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org