You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Mario Corchero (JIRA)" <ji...@apache.org> on 2017/10/07 11:48:04 UTC
[jira] [Commented] (SOLR-10888) almost self-generating python
script(s) to access V2 APIs
[ https://issues.apache.org/jira/browse/SOLR-10888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195659#comment-16195659 ]
Mario Corchero commented on SOLR-10888:
---------------------------------------
I've made some changes to the base script to:
- Use URL + body instead of passing the arguments in the URL (that did not work, commands and parameters are part of the body)
- Add type documentation
- Ability to add new types and the mapping between them
- Defaults for unmappable types/commands.
At the moment the script just prints a curl command to shoot and get the desired output but it can be just executed if we want to.
If this looks interesting there are still a bunch of "TODO" like:
- Supporting non-command actions (just retrieve the list of collections for example)
- Support other endpoints "cores", etc.
- Automagically discover sub-URLs (Specific operations on a collection for example)
- Add more types like array
This is how using it looks like:
{code}
$ ./solr_cli.py --help
usage: solr_cli.py [-h]
{collection:create,collection:create-alias,collection:delete-alias,backup-collection,restore-collection}
...
CLI tool to interact with Solr Admin REST API.
optional arguments:
-h, --help show this help message and exit
collections:
List of commands to manipulate collections
{collection:create,collection:create-alias,collection:delete-alias,backup-collection,restore-collection}
collection:create Create a collection.
https://lucene.apache.org/solr/guide/collections-
api.html#create
collection:create-alias
Allows one or more collections to be known by another
name. If this command is used on an existing alias,
the existing alias will be replaced with the new
collection details.
https://lucene.apache.org/solr/guide/collections-
api.html#createalias
collection:delete-alias
Deletes a collection alias
https://lucene.apache.org/solr/guide/collections-
api.html#deletealias
backup-collection Backup Solr indexes and configurations for a specific
collection. One copy of the indexes will be taken from
each shard, and the config set for the collection will
also be copied.
https://lucene.apache.org/solr/guide/collections-
api.html#backup
restore-collection Restore Solr indexes and configurations from a backup.
You cannot restore into the same collection you took
the backup from. The target collection must not exist
before calling this command, as it will be created by
the restore action. The new collection will have the
same number of shards and replicas as the original
collection, and all routing strategies will be
retained.
https://lucene.apache.org/solr/guide/collections-
api.html#restore
All options and parameters are generated dynamically. Check the Solr Reference
guide for further documentation on the API.
$ ./solr_cli.py collection:create -h
usage: solr_cli.py collection:create [-h] [--config str]
[--router json-object] [--numShards int]
[--shards str] [--replicationFactor int]
[--nrtReplicas int] [--tlogReplicas int]
[--pullReplicas int]
[--nodeSet solr-array]
[--shuffleNodes bool]
[--maxShardsPerNode int]
[--autoAddReplicas bool]
[--rule solr-array] [--snitch solr-array]
[--properties json-object] [--async str]
[--http_method {POST}]
str
positional arguments:
str The name of the collection to be created.
optional arguments:
-h, --help show this help message and exit
--config str The name of the configuration set (which must already
be stored in ZooKeeper) to use for this collection. If
not provided, Solr will default to the collection name
as the configuration set name.
--router json-object These properties define how to distribute documents
across a collection's shards.
--numShards int The number of shards to be created as part of the
collection. Shards are logical partitions of a single
collection. Each shard has at least one replica, but
more replicas for each shard can be defined with the
replicationFactor property. This is a required
parameter when using the 'compositeId' router.
--shards str A comma-separated list of shard names, e.g.,
shard-x,shard-y,shard-z. This is a required parameter
when using the 'implicit' router.
--replicationFactor int
The number of NRT replicas to be created for each
shard. Replicas are physical copies of each shard,
acting as failover for the shard.
--nrtReplicas int The number of NRT replicas to be created for each
shard. Replicas are physical copies of each shard,
acting as failover for the shard. Replicas of type NRT
will be updated with each document that is added to
the cluster, and can use "softCommits" to get a new
view of the index in Near Real Time. This parameter
works in the same way as 'replicationFactor'
--tlogReplicas int The number of TLOG replicas to be created for each
shard. TLOG replicas update their transaction log for
every update to the cluster, but only the shard leader
updates the local index, other TLOG replicas will use
segment replication and copy the latest index files
from the leader.
--pullReplicas int The number of PULL replicas to be created for each
shard. PULL replicas don't receive copies of the
documents on update requests, they just replicate the
latest segments periodically from the shard leader.
PULL replicas can't become shard leaders, and need at
least one active TLOG(recommended) or NRT replicas in
the shard to replicate from.
--nodeSet solr-array Defines nodes to spread the new collection across. If
not provided, the collection will be spread across all
live Solr nodes. The names to use are the 'node_name',
which can be found by a request to the cluster/nodes
endpoint. A special value of EMPTY will create no
shards or replicas for the new collection. In this
case, shards and replicas can be added later with the
add-replica command available on the
/collections/{collection}/shards endpoint.
--shuffleNodes bool Controls whether or not the shard-replicas created for
this collection will be assigned to the nodes
specified by the nodeSet property in a sequential
manner, or if the list of nodes should be shuffled
prior to creating individual replicas. A 'false' value
makes the results of a collection creation predictable
and gives more exact control over the location of the
individual shard-replicas, but 'true' can be a better
choice for ensuring replicas are distributed evenly
across nodes. This property is ignored if nodeSet is
not also specified.
--maxShardsPerNode int
When creating collections, the shards and/or replicas
are spread across all available, live, nodes, and two
replicas of the same shard will never be on the same
node. If a node is not live when the collection is
created, it will not get any parts of the new
collection, which could lead to too many replicas
being created on a single live node. Defining
maxShardsPerNode sets a limit on the number of
replicas can be spread to each node. If the entire
collection can not be fit into the live nodes, no
collection will be created at all.
--autoAddReplicas bool
When set to true, enables auto addition of replicas on
shared file systems (such as HDFS). See
https://lucene.apache.org/solr/guide/running-solr-on-
hdfs.html for more details on settings and overrides.
--rule solr-array Defines rules for where replicas should be located in
a cluster.
--snitch solr-array
--properties json-object
Allows adding core.properties for the collection. Some
examples of core properties you may want to modify
include the config set, the node name, the data
directory, among others.
--async str Defines a request ID that can be used to track this
action after it's submitted. The action will be
processed asynchronously.
--http_method {POST} HTTP Method to use.
$ ./solr_cli.py collection:create example_collection --numShards=1
curl http://localhost:8983/v2/collections/collection:create -X POST -d '{'collection:create': {'name': 'example_collection', 'numShards': 1}}'
{code}
All "known" types are validated. Even complex ones. Example:
{code}
$ ./solr_cli.py collection:create example_collection --numShards=1 --properties="not_an_object"
usage: solr_cli.py collection:create [-h] [--config str]
[--router json-object] [--numShards int]
[--shards str] [--replicationFactor int]
[--nrtReplicas int] [--tlogReplicas int]
[--pullReplicas int]
[--nodeSet solr-array]
[--shuffleNodes bool]
[--maxShardsPerNode int]
[--autoAddReplicas bool]
[--rule solr-array] [--snitch solr-array]
[--properties json-object] [--async str]
[--http_method {POST}]
str
solr_cli.py collection:create: error: argument --properties: 'not_an_object' is not valid JSON
$ ./solr_cli.py collection:create example_collection --numShards=1 --properties="{}"
curl http://localhost:8983/v2/collections/collection:create -X POST -d '{'collection:create': {'name': 'example_collection', 'numShards': 1, 'properties': '{}'}}'
$ ./solr_cli.py collection:create example_collection --numShards=1 --properties="{}"
{code}
> almost self-generating python script(s) to access V2 APIs
> ---------------------------------------------------------
>
> Key: SOLR-10888
> URL: https://issues.apache.org/jira/browse/SOLR-10888
> Project: Solr
> Issue Type: New Feature
> Components: v2 API
> Reporter: Christine Poerschke
> Priority: Minor
> Attachments: SOLR-10888.patch
>
>
> The V2 API supports introspection and the results of such introspection(s) can be used to automatically on-the-fly generate a (nested) {{argparse.ArgumentParser}} in a python script and then to again automatically transform the script arguments into a url and http call to the V2 API.
> Illustrative patch and sample output to follow.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org