You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Markus Jelsma (JIRA)" <ji...@apache.org> on 2012/10/22 16:50:12 UTC
[jira] [Created] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Markus Jelsma created NUTCH-1480:
------------------------------------
Summary: SolrIndexer to write to multiple servers.
Key: NUTCH-1480
URL: https://issues.apache.org/jira/browse/NUTCH-1480
Project: Nutch
Issue Type: Improvement
Components: indexer
Reporter: Markus Jelsma
Priority: Minor
Fix For: 1.6
SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488732#comment-13488732 ]
Markus Jelsma commented on NUTCH-1480:
--------------------------------------
Julien, yes. The Nutch tools are modified to work with a List<SolrServer> now. One or more SolrServers are returned depending on how many you specify comma-separated. Communication to Solr is simply done by iterating over the List and repeating the commands such as add or delete or commit.
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488744#comment-13488744 ]
Markus Jelsma commented on NUTCH-1480:
--------------------------------------
I think you mean the justed linked issue NUTCH-1377? I just happen to work on that issue. Using the CloudSolrServer will send the docs to the correct shard already. The way it works now is sending millions of records to a single node which then distributes it again, a waste of IO. That issue will work with this issue so i may want to push them in together. It's just a matter of returning the correct SolrServer instance and working around the HTTPCLient issues.
agreed on deduplication. It would also be hard for this issue to work with dedup because not all indices may be identical.
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Lewis John McGibbney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490316#comment-13490316 ]
Lewis John McGibbney commented on NUTCH-1480:
---------------------------------------------
A suggestion fore the dedup job is to read in the comma separated solr cores/servers before dropping everything after the first server within the list. Some simple logging could then indicate that dedup was only attempted and subsequently executed on the first server...
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493163#comment-13493163 ]
Markus Jelsma commented on NUTCH-1480:
--------------------------------------
I'm fine with that too, i don't really care as long it's obvious enough for the user. Any more thoughts to share? Opinions?
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma reassigned NUTCH-1480:
------------------------------------
Assignee: Markus Jelsma
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on NUTCH-1480 started by Markus Jelsma.
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488728#comment-13488728 ]
Julien Nioche commented on NUTCH-1480:
--------------------------------------
Hi Lewis
bq. Can I run multiple Solr servers in psudo distributed mode?
SOLR is completely separated from Hadoop and has nothing to do with local vs distrib. You can run serveral instances of SOLR on the same machine if that is your question. Just invoke a different port when starting it from the command line with a separate SOLR home.
Markus,
Just to make sure I understand - this sends ALL the documents to ALL the SOLR instances specified, right?
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488738#comment-13488738 ]
Julien Nioche commented on NUTCH-1480:
--------------------------------------
OK thanks. What about having a mechanism for specifying a way of distributing the docs with the replicate-to-all being one of the options? Could do consistent hashing maybe? I expect that most people would want to shard.
off topic re-deduplication : I think we've hit the limits of the current mechanism which I assume was based on the one we had when Nutch was managing its own Lucene indices. It's not reasonable to pump ALL the docs from SOLR into Hadoop to dedup and I'd rather have map reduce jobs to find the duplicates based on the crawldb and send the deletion commands to SOLR. And this would work for ElasticSearch as well. Am pretty sure there is a JIRA for this somewhere
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work stopped] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on NUTCH-1480 stopped by Markus Jelsma.
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Lewis John McGibbney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488721#comment-13488721 ]
Lewis John McGibbney commented on NUTCH-1480:
---------------------------------------------
Hi Markus. Can I run multiple Solr servers in psudo distributed mode? If so can you provide a link the a (solr) wiki entry and I will try this out. Also can you clarify between core and servers? I don't see the server terminology listed [0] on the Solr wiki. Thanks
[0] http://wiki.apache.org/solr/SolrTerminology
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Lewis John McGibbney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488852#comment-13488852 ]
Lewis John McGibbney commented on NUTCH-1480:
---------------------------------------------
Mmm. Additionally there is a problem with this when you run a continuous or large crawl with the crawl script. By default it attempts to dedup and the job fails. Funatemtally though, the patch works with trunk (and Solr 4.0) as you describe... after I've applied the patch in NUTCH-1486
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488730#comment-13488730 ]
Markus Jelsma commented on NUTCH-1480:
--------------------------------------
Hi Lewis, Solr does not have a notion of pseudo distributed mode. You can simply run one server with two cores, two servers on different ports or different machines or have a SolrCloud cluster in multiple NOCs, all have a unique URL. A SolrCloud cluster is roughly the same as multiple stand-alone servers working together.
A core is similar to a table in a SQL server.
See http://wiki.apache.org/solr/CoreAdmin#Example on how to quickly set up two cores in a single Solr server. Copy Nutch' schema and you have two Nutch indices you can write to in a single go.
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Markus Jelsma updated NUTCH-1480:
---------------------------------
Attachment: NUTCH-1480-1.6.1.patch
Patch for 1.6 allows the SolrUrl parameter to be a comma-delimited list of URL's. This works for the indexer and clean tools, not the deduplicator. It also slightly improves logging, one can now see how many docs were added or removed.
Any comments?
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Julien Nioche (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488786#comment-13488786 ]
Julien Nioche commented on NUTCH-1480:
--------------------------------------
nope. I meant implementing the distribution to the shards on the Nutch side without relying on the CloudSolrServer. Having said that we want to move to SOLR4 and if we get that from SOLR for cheap then that's even better
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (NUTCH-1480) SolrIndexer to write to multiple
servers.
Posted by "Markus Jelsma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/NUTCH-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488856#comment-13488856 ]
Markus Jelsma commented on NUTCH-1480:
--------------------------------------
Jul, yes, doing it via SolrJ and Zookeeper is much cheaper. We don't need to obtain hash ranges ourselves.
Lewis, didn't think about that. That's a problem indeed. Although it is technically possible we must not dedup with two indices at the same time, it's going to be trouble. Even with this patch it will still work until someone decides to use the crawl command with two Solrservers comma-separated. Although i don't believe users of the crawl command will write to multiple locations we must prevent them from doing so. Perhaps a simple check for a comma in the solr URL and an exception?
Something else? Drop deduplication altogether and rely on Solr's internal deduplication (which doesn't work for clouds though)? Other clever ideas?
> SolrIndexer to write to multiple servers.
> -----------------------------------------
>
> Key: NUTCH-1480
> URL: https://issues.apache.org/jira/browse/NUTCH-1480
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.6
>
> Attachments: NUTCH-1480-1.6.1.patch
>
>
> SolrUtils should return an array of SolrServers and read the SolrUrl as a comma delimited list of URL's using Configuration.getString(). SolrWriter should be able to handle this list of SolrServers.
> This is useful if you want to send documents to multiple servers if no replication is available or if you want to send documents to multiple NOCs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira